I am currently re-writing the portion of the code that involves reading the files that have been downloaded and deciding what are the next set of files to download. This involves reading file names in a directory and sorting them in descending order so that I can find out what was the most recent date. I quickly ran into the problem during the sort because dates like “2013_12_2” were placed right beside “2013_12_20” and the only way I can sort it properly is if I put them into “date” objects. Hence I decided to move away from using fopen() on files and store everything into a database instead. This will also help in optimise performance when I am reading and writing values using database.
- It had natural built in support for Python syntax in object form.
- It is free.
- Came pre-installed with Anaconda
- Natural support for DateTime objects
- Uses ViTables to view the tables.
SQLite has the following advantages :
- Uses SQL-like query syntax which is something of an industry standard
- Comes pre-installed with Anaconda
- With v3, it supports Date and DateTime (Timestamp) columns
- Quite a couple of software that can be used to view the tables. I decided to use SQLiteStudio.
So in the end I decided to go for SQLite because it allows me the chance to familiarise the SQL Syntax, plus the datatype support is sufficient for me to move forward and SQLiteStudio software on Mac OS X works good! Supposedly ViTables should work fine on any platform but there was no straightforward way of installing the software on Mac OS X. I guess the disadvantage of not using PyTables is that I will miss out on using NumPy or Pandas.