All Posts

The Oxfordshire Science and Ideas Festival 2018 starts today, running from 12th to 22nd Oxford. It’s an annual festival, organised association with Science Oxford and the University of Oxford, to bring science and ideas from the labs, colleges and lecture theatres of Oxford to engage with the public and build relationships. It has a phenomenal range of events and this year the Big Data Institue is diplaying some of its work through a series of posters ‘Big Data, Big Future!’ in the middle of Oxford, outside the Business School, passed by tens of thousands of people each day on their way to and from Oxford train station. It was a real privilege to be asked to create a poster for this exhibition, and to show some of the work that we’re doing on supermarket shopping, nutrition and public health, and you can see it both in person at the Business School, and also below. I passed the exhibition this morning and they all look even better in person, and as a collection, illustrating projects as diverse as using brain MRI images to better understand Alzheimer’s disease and how it could be predicted, investigating the relationships between traffic pollution and health, using wearables for accurately monitoring physical activity, creating long term evidence about the development of dementia through people playing games on your phone, and tracking the spread of malaria and efficacy of various efforts to combat the disease.

Django is designed to work with a Django-specific database, that the framework builds and manages through the core Django models (e.g. for authentication) and the data models that you create. It’s a fantastic system, and has been improved massively by including migrations in the Django core from version 1.7 (previously migrations were best dealt with by the fabulous South package developed here in Oxford by the friendly folk at Torchbox). However, sometimes the occasion arises where you want to use an existing database with Django, which is a little trickier, but after combining digging around a few resources, plus some testing, is actually pretty straightforward.

The implementation of data science and big data is transforming the worlds of technology, finance, entertainment, retail, travel, sport and healthcare. In our research group at the University of Oxford, we’re applying them to public health research, using data science and big data to find big answers to some of the big questions and problems that affect us today. We’re looking at population-level approaches to diet and nutrition research, specifically working on three exciting projects related to analysis of online supermarket data, and how to have an impact on public health.

PostgreSQL has some amazing features that make it a brilliant choice as a database engine. As well as being a fully functional RDBMS, it is ACID compliant, supports JSON fields, can support GIS, and, of course, it’s open source, completely free, and has a great community behind it. However, a couple of things weren’t as easy as I’d hoped, namely working with postgres data on an external drive, and also dealing with different versions of postgres and fixing by upgrading. After going through both of these recently, I thought I’d write down the procedures as a a reference.

Data transfer between JavaScript client and server is made relatively simple by using the XMLHttpRequest API.

Data transfer between JavaScript client and server is made relatively simple by using the XMLHttpRequest API.

The package manager Homebrew is a phenomenal tool which has made working on a Mac even easier and more productive. It’s a great way to keep tools and packages updated, is easy to manage, and my time spent fixing dependency issues and similar problems has been greatly reduced ever since I started using it a couple of years ago. However, as Homebrew doesn’t delete old versions of installed formulae, over time the ‘Cellar’ become filled with old/unused files and directories and it was time for a clearout. As with most things involving Homebrew, this was very straightforward and relatively painless:

Environmental variables are a great way to dynamically set credentials and other variables, which helps to increase portability of projects between systems. The excellent virtualenvwrapper, which helps when setting up and using python virtual environments, makes setting these environmental variables a trivial process.

Setting up batch jobs for regular housekeeping tasks can be easily handled using shell scripts. This is an example of some code that could be used for backing up a number of databases with a single script, and recording those backups in logfiles.

Last weekend I helped a couple of teams during the Digital Health Oxford hack weekend (which I was running, and so couldn’t commit to a single project/team). The contribution to each team involved doing a some data mainpulation, extracting, loading etc, and one required using scraping using Selenium, which I’d dabbled with before but never used for anything specific before. Whilst a writeup of the weekend and projects will be on the DHOx website, I wanted to write up a short description of the work I contributed to here.

Jupyter notebooks (formerly iPython) are becoming an ever useful tool for many people, especially those working in a scientific field. There are a number of specific techniques that enhance their utility, and I’ll try to post details of these over the next few weeks. One useful tip involves how to use variables within widgets in notebooks.

The ever-excellent Freakonomics podcast published a fantastic episode a couple of weeks ago on ‘How to Be More Productive’. One of the many great things about it was a list of 8 key tools/skills that kept coming up for Charles Duhigg during research for his book, Smarter, Faster, Better:

I’ve found the need to print a lot of JSON for recent projects. Examining the contents of a JSON, useful for initial eyeballing of the data, can be tricky as the content and formatting can be quite dense. However, there are a couple of tools that I’ve been using to make this easier.

Where do I put that alias? What about the terminal prompt configuration? Which config file does this belong in?

Date and datetime fields are incredibly useful in MySQL databases, but can be a little tricky to use. Here are a few tips for common tasks

A common task is to manipulate date and time variables in Python, and get them to interact nicely with a database - I’ll describe the use with MySQL here.