Posts

Showing posts from March, 2020

Flip images in multiple directories for data augmentation with no effort

Image
When preparing a dataset for training a neural network, You would like to fully exploit that dataset to increase your network's accuracy as many papers provided evidence that some of these methods do improve the network's recognition ability, Some of those methods are flipping the images horizontally and scaling the images, all while keeping the original images unscathed. Picture from the oppenheimer's famous words video. Assuming you are using linux or macOS, First you will need to install graphics magick which is super simple sudo apt-get install graphicsmagick we will use the mogrify function from graphics magic to flip and resize images. import subprocess def prepare_dataset():     DIR_PARENTS = ['dir1/',                                     'dir2/',                             ...

Restore corrupted innodb from .frm and .ibd files

Sql comes in many different,  for some flavours, if the db crashes, updated or anything is changed within it. You can simply move the files in the directory of your new sql db and it should read it instantly. With innodb , it does read them but when you try to SELECT from the tables, it returns TABLE does not exist. First install mysqlfrm which we will use to read the tables form. sudo apt-get install mysql-utilities Next do mysqlfrm --user=mysql --server=root:maria@localhost --port=3308 sql_table.frm You can change the variables with the ones specific to your server of course. The output of this will be a command that you can use to create a table of the same form. Create a new db of a name that you desire,  then copy the output of mysqlfrm and execute it to create a table inside our newly created db. Now that a table of the same form is created, Execute those commands ALTER TABLE 'sql_table' ROW_FORMAT=DYNAMIC; ALTER TABLE sql_table DISCARD TABLES...

Create a route optimization algorithm with zero costs using google's OR-tools and OSRM Part 4

Image
We continue with our series  and we create a function that takes care of reading the output of the algorithm, converting it to a readable form and plotting the routes on the map. def print_solution(data, manager, routing, assignment):       Route = []       for vehicle_id in range(data['num_vehicles']:                 index           = routing.Start(vehicle_id)                 idx               = manager.IndexToNode(index)                 plan_output = 'Route for vehicle {} with ID {} and capacity {}:\n'.format(vehicle_id, data['IDs'][idx], data['capacity'][vehicle_id])                 Route[Iter].append([data['locations'][idx], data['IDs'][idx]])                 prev_ind...

Create a route optimization algorithm with zero costs using google's OR-tools and OSRM Part 3

This is a continuation to the route optimization series . First we define the function that will be used by the optimizer to get the distances between points.   def distance_callback(from_index, to_index):                 from_node = manager.IndexToNode(from_index)                  to_node      = manager.IndexToNode(to_index)                          if from_node == to_node:                                return 0                          return data['distance_matrix'][from_node][to_node] IndexToNode is used to correlate between a location as a node index inside the being tested/optimized route by the solver and the index of the loc...

Create a route optimization algorithm with zero costs using google's OR-tools and OSRM Part 2

We have covered querying the server for data about events, offers and requests in the first tutorial . Now we start building the data to be fed into OR-tools. Offer_df      = False Request_df  = False while type(Offer_df)     == bool or           type(Request_df) == bool:                 df = read_events()                 Offer_df = read_offers(df['event_id'])                 if type(Offer_df) != bool:                        Request_df = read_requests(Offer_df['event_id']) The reason why we check for the type of Offer_df or Request_df  instead of checking if they are equal to False is that if Offer_df is a numpy array, then python will return an error as comparing a numpy array to a value is invalid and it will ask us to be more ...

Create a route optimization algorithm with zero costs using google's OR-tools and OSRM Part 1

Image
This is a tutorial for an algorithm that has riped over the years of experience in doing algorithms and websites  for carpooling, Uber like service, Freight routing, and many other applications. I will be stating the current algorithm and i will discuss my decisions on certain code implementations as well as some of the previous decisions that i took and proved to be wrong, slow or error prone. This algorithm is capable of optimizing for thousands of locations, and made to be highly customizable. Features of the algorithm are: Reading data from and exporting data to an excel sheet or a Database. Multiple time windows per location. Multiple vehicle types, capacities, buffers, depots. Custom optimization parameters like distance, time, fuel consumption, etc. Pickup and dropoff. Custom loading and unloading time per stop. Custom service time at certain locations. Custom max idle time waiting for the next stop's time window. Custom distributed workload amongst ...