Back in June, 2013 I wrote a post that was primarily focused on the crime categories that I had put together to describe all the various crimes reported by OPD. Part of that analysis included a comparison against CrimeMapping.com ’s resource. Back then I said:
An experiment was performed taking data provided by OPD for the dates 18 – 30 April, 2013. OPD listed 1167 incidents during this period. CrimeMapping reports a total of 496 during this same period, i.e., only 43% of what OPD provides!
Recently, an otherwise well-informed person said: “I thought CrimeMapping.com had fixed those bugs.” I went back to check. Bottom line is that the results now are about the same:
CrimeMapping.com is now dropping 70% of the data OPD reports elsewhere! On top of that omission, they are losing information in their naming of crimes that OPD has described more completely. Both are useful lessons.
Background
CrimeMapping.com is an interface provided by Omega as part of a suite of “CrimeView” analytic tools sold to OPD. One product is designed for use by top-level police captains, a more constrained version is for operational use by crime analysts within OPD, another version for use by city council members and other trusted users. Omega sells these solutions to many other police departments (the CrimeMapping.com home page lists 370 different agencies), and so Oakland’s specifics are “averaged” into the demands of Omega’s larger market; as demonstrated below, that turns out to matter!
CrimeMapping.com is the public face of Omega’s products, primarily to provide map-based viewing. It benefits from its special relationship with OPD to have access to data bits (e.g., perhaps the statute codes associated with crime incidents) not publicly available. CrimeMapping.com mapping functionality makes it redundant with OPD data provided via Oakland’s newer Socrata interface. (Given the importance of the mapping facilities, it is also worth noting that address information provided by OPD elsewhere is missing in both CrimeMapping.com and Socrata streams!)
Experimental details
I wanted a data sample that was recent, but not so recent that lags in reporting might affect the data CrimeMapping.com had to display. I chose January 2014.
But CrimeMapping.com is designed to report only ~500 data points at a viewing. Doing my best using CrimeMapping.com ’s drop-down list/interface of agencies to select just Oakland ’s reports (see below) and beginning with January 1, 2014, various end dates were tried. Finally, January 10 generated 564 incident reports.
CrimeMapping.com does not provide an export facility, so a crude parser/screen-scraper was used to turn their HTML into a data-rich CSV file. In the process it became clear that despite the query focusing on OPD, CrimeMapping.com had included 155 records from Alameda’s police department?! Culling those leaves a remaining set of 377.
Dropped data
During the period Jan 1-10 2014 (inclusive) OPD generated 1193 crime reports — defined in terms of unique “casenumber” (aka RD#) — concerning 1295 incidents. Using matching case number and date-time stamps as the basis of correspondence, the intersection of the two sets included 365 cases. There were 829 OPD cases (896 incidents) that CrimeMapping.com did not mention, and 12 cases CrimeMapping.com mentioned that OPD did not.
Certain categories of crimes seem particularly likely to be dropped by CrimeMapping.com:
Type : Freq
BURG – AUTO : 165
STOLEN VEHICLE : 92
PETTY THEFT : 74
MISDEMEANOR ASSAULT : 61
BURG – RESIDENTIAL : 47
VANDALISM : 38
GRAND THEFT : 24
Here is a simple CSV file (same format as OPD’s) with several examples of these crime types, reports by OPD but not included by CrimeMapping.com ; try them yourself!
Crime type mismatches leading to information loss
A deeper, more difficult issue concerns mismatches between the various categories used by OPD contrasted to what CrimeMapping.com shows. Recall that CrimeMapping.com is a product sold by Omega across the country to many other police departments. Omega’s corporate goal must be to minimize the amount of time and effort they spend massaging any one city’s data into their standardized interface. They have some logic which takes a data stream similar to (but probably more informationally rich than!) the one provided to Oakland’s citizens, and converts it into the categories they show on their maps.
Comparison of OPD crime categories vs. those from CrimeMapping.com provides multiple examples where OPD is providing a richer language for crime types.
EG, CrimeMapping.com’s crime “category” of THEFT-LARCENY is distributed across multiple OPD crime types:
OPD Category : N
PETTY THEFT : 35
BURG – AUTO : 28
GRAND THEFT : 10
BURG – OTHER : 1
EG, CrimeMapping.com’s “description” of LARCENY THEFT (EXCEPT MOTOR VEHICLE THEFT) is distributed across multiple OPD crime types:
OPD Description : N
THEFT : 31
BURGLARY-AUTO : 28
GRAND THEFT : 6
THEFT PERSONAL PROPERTY/PETTY THEFT : 2
GRAND THEFT:FIREARM/ANIMAL/ETC : 1
ATTEMPTED BURGLARY-AUTO : 1
SHOPLIFT : 1
GRAND THEFT:MONEY/LABOR/PROPERTY OVER $400 : 1
SC GRAND THEFT : 1
OBTAIN MONEY/ETC BY FALSE PRETENSES : 1
GRAND THEFT FROM PERSON : 1
CrimeMapping.com is losing information regarding these distinctions.
Relating OPD’s major (“Part 1”) crime categories to national the FBI’s Uniform Crime Reporting (UCR) system is done every month; better reporting standards like National Incident-based Crime Reporting (NIBRS) are also available. There is no good reason Oakland residents should have their data “dumbed down” to categories that happen to have processing efficiencies for private vendors like CrimeMapping.com.