O'Reilly and Associates, one of the best recognized technical book producers, conference hosts and online publishers, featured an article I wrote about automating a web photo gallery for my favorite photographer, who also happens to be my wife. The article describes using iPhoto, Email, Sendmail, ImageMagick, MySQL and Perl to handle moving thousands of images from an image management system to a web-based photo gallery. The article was later included in O'Reilly's Mac OS X Panther Hacks as Hack #45.

Georgetown Loop Railroad

Hot Air Balloon

Georgetown Loop Railroad

An environmental consulting company needed to retrieve data from a corporate website every month to check their costs against their tasks. The company resorted to hiring temporary workers who would access several thousand web pages and print them out at the beginning of each month. The agents would then examine the printouts and compare the results.

Three separate applications were built to address the company's needs. One application accesses the web site each week (pulled down 16,317 dynamic web pages on 08/18/2008, totalling 2,785,093 records in one table and 7,384,092 records in another since 01/19/2004), extracts specific sections of the web pages and associated budget data, then stores the information in a database.

The second application allows the agents to view the information the printouts contained in a streamlined view, without the multiple steps that were required to navigate the original website and the extraneous information. A simple interface, tailored to their requirements allows them to select from one record to thousands of records based on a variety of criteria. The results can be viewed directly, or emailed for later access.

The third application allows the agents to view the budget data in a variety of ways. An HTML table presents a quick view with buttons at the top of each column to provide sorting in either direction. An Excel file can be generated comparing the budgets for all projects over different dates. And a Java-based graphing package provides details of a specific project over a range of dates.

A scientific software development company needed to support a format for satellite data processing. The format did not contain some necessary parameters that were needed by the satellite community to perform typical processing tasks. Users were required to visit two US Geological Survey sites where they had to download files and extract data from each file.

The two USGS files were identified as a parameter file and a record file. The parameter file had 1500 lines, of which about 20 lines were necessary for the task. The record file had over 300,000 records in a format only a database could love. Information from the satellite data was used to select the proper record from the database file, which was then used to select the fields from the parameter file. The parameters were then inserted into the application GUI to perform the processing tasks required.

Several applications were needed to accomplish the goals. One simply downloaded the latest files from the USGS sites on a weekly schedule. The second application pulled the sets of data needed from the parameter file and placed them in a database table. The third application compared the records in the large database file to the local database and added only the changed records. A CGI application was built that would respond to a URL with multiple parameters based on the satellite data file. Finally, the last application accepted the information from the CGI application and populated the GUI of the scientific software application.

A video recording task was in progress at a defense contractor that required hundreds of thousands of frames to be recorded to disks for a weapon simulator. The existing process required three shifts of two people each to monitor the recording and to enter data every hour or so to format the next batch of frames and verify the last batch of frames. The data entry was technical and error-prone since the entries consisted of large groups of numbers entered as commands to the recording program.

After some evaluation and development time, several programs were developed to interact with the recording system and to evaluate the results. After several days of excellent performance the three shifts were reduced to one, and eventually reduced to a single employee verifying the results for a couple of hours each day.