I am currently re-writing the portion of the code that involves reading the files that have been downloaded and deciding what are the next set of files to download. This involves reading file names in a directory and sorting them in descending order so that I can find out what was the most recent date. I quickly ran into the problem during the sort because dates like “2013_12_2” were placed right beside “2013_12_20” and the only way I can sort it properly is if I put them into “date” objects. Hence I decided to move away from using fopen() on files and store everything into a database instead. This will also help in optimise performance when I am reading and writing values using database.

So I spent the last 2 weeks researching between PyTables and SQLite and finally settled on SQLite. At first I was learning towards PyTables because :

  1. It had natural built in support for Python syntax in object form.
  2. It is free.
  3. Came pre-installed with Anaconda
  4. Natural support for DateTime objects
  5. Uses ViTables to view the tables.

SQLite has the following advantages :

  1. Uses SQL-like query syntax which is something of an industry standard
  2. Free
  3. Comes pre-installed with Anaconda
  4. With v3, it supports Date and DateTime (Timestamp) columns
  5. Quite a couple of software that can be used to view the tables. I decided to use SQLiteStudio.

So in the end I decided to go for SQLite because it allows me the chance to familiarise the SQL Syntax, plus the datatype support is sufficient for me to move forward and SQLiteStudio software on Mac OS X works good!  Supposedly ViTables should work fine on any platform but there was no straightforward way of installing the software on Mac OS X. I guess the disadvantage of not using PyTables is that I will miss out on using NumPy or Pandas.


Author: Zac

Think & Do

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s