Labordata should be easier to use and much snappier. Unfortunately, some older saved queries may not work because of changes between the SQL dialect. Please get in touch if you need help or have run into any other problems.

Dialect Improvements

SQLite’s minimalist SQL dialect and loose typing can lead to complicated queries.

Here are two queries that both get the upper and lower bound on count of LM20 contracts filed per year. The first is in the SQLite dialect:

with most_recent_amendment as (
  select
    *
  from
    (
      select
        *
      from
        filing
      order by
        srNum,
        beginDate,
        endDate,
        formFiled,
        yrCovered,
        amendment desc
    )
  group by
    srNum,
    beginDate,
    endDate,
    formFiled,
    yrCovered
),
new_contracts_lower as (
  select
    yrCovered as year,
    sum(repOrgsCnt) as "new contracts minimum"
  from
    most_recent_amendment
  where
    formFiled = 'LM-20'
  group by
    yrCovered
),
new_contracts_upper as (
  select
    yrCovered as year,
    sum(repOrgsCnt) as "new contracts maximum"
  from
    filing
  where
    formFiled = 'LM-20'
  group by
    yrCovered
)
select
  *
from
  new_contracts_lower
  inner join new_contracts_upper using(year)
where year <= 2025
order by
  year desc;

DuckDB is built for analytic queries and so its dialect is much nicer for this type of query.

WITH ranked AS (
  SELECT
    *,
    row_number() OVER (
      PARTITION BY srNum, beginDate, endDate, formFiled, yrCovered
      ORDER BY amendment DESC
    ) AS rn
  FROM filing
  WHERE formFiled = 'LM-20'
)
SELECT
  yrCovered AS year,
  sum(repOrgsCnt) FILTER (WHERE rn = 1) AS "new contracts minimum",
  sum(repOrgsCnt) AS "new contracts maximum"
FROM ranked
GROUP BY yrCovered
HAVING year <= 2025
ORDER BY year DESC

Speed

As for speed, on analytic queries, we have been seeing 10-50× speedups.

I’ve not priortized this data previously, but I had to do some major refactors of the scrapers of the LM 10, 20, and 21 data (which are now working again), and after those changes it was easy to add the LM 30 data, so I have.

Speaking of the LM 20 and 21 data, I was thrilled to see that data used by the Economic Policy Institute and LaborLab in their report estimating that union avoidance is a $1.5 billiion dollar industry.

If you have used any data from the Labordata project for external publications or internal analyses, I would be very grateful if you could share that with me!

labor-union-parser is a new Python package to lookup the a local union’s Office of Management and Labor Standards (OLMS) filing number from short texts like “United Automobile, Aerospace and Agricultural Implement Workers of America, UAW Local 1803” and “LOCAL 6, NEW YORK HOTEL & MOTEL TRADES COUNCIL, UNITE HERE.”

Most labor organizations representing private sector workers are required to file an annual financial report with the Department of Labor’s Office of Management and Labor Standards. The Office maintains a more or less consistent filing number for unions across the filings, and so office’s data is the closest we have to a comprehensive gazette for private-sector-representing labor unions.

This tool uses a probabilistic model to do the lookup, and it’s pretty accurate. It predicts three things, is a text referring to a union, what is the union affiliation of the local; what is the filing number of the union.

In our test set, for the records we were able to identify as referring to particular filing number we get 98.3% accuracy. As we’ll see the performance is often better than that.

Practical example

I have this wired up into labordata.bunkum.us, where it runs every night against any column in any table that refers to unions. Here’s a query of the number of distinct election petition cases that unions participated in the last quarter of 2025.

Across the 288 variations of the texts for participants, there is only one error, and an instructive one. The California State University Employees Union is mislabeled as belonging to “Field Representatives Union,” the staff union of the California Federation of Teachers.

To this point the California State University Employees Union has been a public-sector union, and have never filed with the OLMS. This tool will go badly wrong if the texts include public sector unions. If you have ideas for a good data source for public sector locals, please let me know.

Training data

In the repo, I have put together a training corpus of over 150,000 examples.

The accuracy of this model is the best I’ve been able to achieve, but much smarter folks than me read these notes. Please use the data to make a better model.

Methodology #

Labor unions with annual receipts larger than $250,000 are required to fill an LM-2 form and submit it to the Department of Labor’s Office of Labor-Management Standards yearly.

We use this form to find unions that are using a CRM product and how much they are spending on it.

Here’s our procedure. For each local union that files LM-2 forms, we look at the forms filed starting in 2010 for mentions of the CRM company or product in the disbursement or purchases schedules (schedules 18 and 4, respectively). If a union has multiple years in which a CRM company or product were mentioned, we just look at the most recent form. If more than one company is mentioned, we choose the company that received more money from the union.

This method definitely under-counts the number of unions that are using a CRM product and how much they are paying for it, but it is difficult to evaluate the extent of the error. In June, 2022, UnionWare claimed to serve 158 unions across three countries. Our methodology has finds 43. If their number is still accurate, then our estimate for UnionWare is off by at most 267%.

The CRM companies and products are listed below:

Notes:

A number of other vendors do sell CRMs, but their CRMs are much less popular than their other offerings. Since the LM data typically does not allow us to identify which product was bought from a vendor, we exclude these vendors: Microsoft, Oracle, and RCS Union Software.

If we segment the locals by size into three groups (small: 0-999, medium: 1,000-9,999, large: 10,000+), we can see that SalesForce’s and UnionWare’s spend is driven by the large locals.

Here is the underlying data behind these charts:

Transcribed County Table for CHIPS dataset

Sean Brailey, Contract Data Engineer, Labor Education and Research Center, transcribed the l_county table from the codebook for the CHIPS dataset.

The CHIPS dataset contains information about NLRB cases from 1984-2000, and with Sean’s work we can now quickly see the top five counties for RC cases in those sixteen years: Cook, IL; Los Angeles, CA; New York; NY; Wayne, MI.

Previously, we had the addresses of cases, so this type of analysis was possible before, but with much more effort.

FMCS FOIA deadend

It looks like our long effort to try to get more unique identifiers for unions and employers for the F7 Intent to Bargain notices has run its course. According the the FOIA officer, the fields we want from their case management system are not used even though they are present in their database schema.

So, back to record linkage.

We can use the information from these forms to make a measurement of how often the leadership of local union changes.

We would like to know the rate at which presidents of a local union turns over. We can approximate the rate of changes in presidents by counting the number of distinct last names that are listed for a president in a yearly filing divided by the number of yearly filings submitted by the union. In order to correct for biases we subtract one from both the top and the bottom.

$$\text{Turnover} \sim \frac{\text{Distinct names} - 1}{\text{Years} - 1}$$

If the the last name is very common, or if presidency passed from one relative to another, then our measure will underestimate turnover. If the filing has a misspelling of the president’s last name, then we will overestimate.

Generally, though, our measure seems to be a reasonable approximation of presidential turnover.

One way, we can check is to see if our measure performs as we would expect from the literature on local union democracy.

In J.C. Anderson’s 1979 article, “A Comparative Analysis of Local Union Democracy”, Anderson identified a number of factors that are correlated with measures of local union democracy. In particular, he found that larger unions and long-established unions tended to have less competitive elections for officers.

Competitive elections for union officers should have a strong relation with turnover of officers. Losing elections is one direct cause of turnover. Similarly the fear of losing, or unwillingness to fight a difficult campaign also lead incumbent officers to decide to not run again.

Union Size #

As expected, larger local unions do have lower turnover of presidents.

Age of Union #

And younger locals do have higher presidential turnover.

This chart only include locals started after December 1, 1970. The data from the OLMS does not include establishment data for local unions established before that date.

Conclusion

At least in aggregate, our measure of presidential turnover seems reasonable. All the relationships shown here are still substantial and statistically significant in a multiple regression analysis.

It could be further refined by stricter checking that a difference in last names means the person is different or the sharing of last names means the person is the same. Using first name and a fuzzy string distance to allow for some typos would be good next steps.

For modelling, logistic regression is likely the most appropriate choice. Every year would be a trial with a binary outcome of whether this years president is the same or different than last year. This modeling choice would handle different number of available years naturally. This modeling would also allow for year-varying features.

Suggestions #

From twitter, I got a number of really useful suggestions. Where the tweet was public or I have permission, I will cite the user.

First, someone rightfully pointed out that presidents can change while the faction in power remains the same. It could be really useful to compare the entire set of officers in one year to the entire set officer in the next. If many or all officers changed, that would be a much better signal that the union has meaningfully different leadership than if just the president changed. I think this idea could really improve the precision of the measure.

One way you could implement:

$$\frac{\text{Distinct Officer Names} - \text{Distinct Officer Positions}}{(\text{Years} - 1) \cdot \text{Distinct Officer Positions}}$$

This is still a bit mushy. Would not distinguish a pattern of officers change that has a high level of continuity

versus many years of complete continuity and then a sudden change.

Second, Dave Kamper suggested that turnover should correlate with:

• salary of president (I bet this correlates strongly with size of union)
• whether the local has paid staff
• the financial trajectory of the union
• the membership trajectory

Appendix #

Employers who hire labor relations consultants (union-busters) or who have financial dealing with union officials have to file a LM-10 form with the Department of Labor disclosing that activity.

The database now has these LM-10 filings, updated nightly.

These are a good complement to the LM-20 and LM-21 forms that we already scrape nightly.

One shortcoming of these forms is that they are all at the firm level. They do not disclose information about the specific establishment where the employers are trying to suppress unionization.

It would be good to get these forms amended so they did include that information.

Voluntary Recognitions

We have added the NLRB voluntary recognitions notifications for the second and third quarter of 2023.

The NLRB was planning on rescinding the Trump-era rule that the NLRB had to be given notice of a voluntary recognition in order for union to enjoy a period of protection from an NLRB certification election. It looks like the new rule was supposed to be finalized this fall, but for some reason it looks it has not. Does anyone have an idea about why not?

I’ll keep on asking for these notices quarterly, until the data stops.

NLRB Participant Addresses

The NLRB’s website lists participants in a case, but does not provide the full address of the employer and unions involved, just the city, state and zip code.

I submitted a FOIA request, and now we have the full address for participating. employers and unions for all case in the NxGen system through September 15, 2023.

These data are useful for identifying the particular establishment of the employer, not just firm, and for figuring out which union local was involved in an NLRB case.

I have updated the participant table of the nlrb database with these more complete addresses.

In the course of making these improvements, I found that I wasn’t fully capturing the role of participants (petitioner, intervener, etc), and that I was even missing some participants in the database. The scraping schedule has been adjusted to prioritize fixing these data issues, and they should be addressed in a few weeks.

Note: the names and addresses of individual persons who are parties to NLRB cases have been redacted.

Jeremy Singer-Vine published a public records request response from the Federal Mediation and Conciliation Service (FMCS) that provides the schema for the service’s case management system.

These records show all the different tables (i.e. spreadsheets) that the case management software uses to tracks data. For each table, the records list the column names.

Using a schema, you can get a pretty good idea of all the information that the FMCS tracks digitally.

I was very interested in it because it appears that the at least employers and maybe bargaining units have unique IDs in their system.

That would mean it would be possible to get longitudinal dataset of bargaining units from the record of F-7: Notice of Intent to Bargain forms.

I’ve thought about that dataset for a while and have started on the fuzzy matching to try to build it, but if we can get it straight from the FMCS that would be much better.

If you know anybody that has familiarity with the FMCS’s data systems, it would be great to talk to them on background. If you have ideas, please let me know.

NLRB NxGen case management schema

Looking through the data exports from the previous NLRB case management systems, there’s a lot of interesting fields that are not available for current cases through the NLRB website (industrial classification is the one that makes my heart ache).

I put together a public records request for the schema of the NLRB’s NxGen case management system, and they fulfilled it.

The case information that is available on the NLRB website is a subset of the data in the NxGen system, and so the schema can be a useful guide to finding information that the NLRB tracks but does not routinely publish. Much of that unpublished data should be accessible through a public records request.

First Quarter of NLRB filings

The first quarter of 2023 looks a lot like the first quarter of 2022. While the scale of organizing is still not near enough, NLRB organizing appears to be continuing at a levels we haven’t seen since the 2000s.

In 2022, there were 209 voluntary recognitions known to the NLRB covering at least 11,182 workers. The caveat of “known to the NLRB” is important because there is no requirement that voluntary recognitions be reported to the NLRB, and we have no idea about what proportion of voluntary recognitions are ever reported to the NLRB.

That said, in 2022, there were about 1,100 NLRB certifications that resulted in certified bargaining representative covering over 55,000 workers.

Understanding organizing activity outside the NLRB process remains a priority.

The future of this NLRB data is not certain as the NLRB is considering returning to the previous voluntary recognition bar rule, which prevents the filing of representation election petition within six months of a voluntary recognition. In 2020, Trump’s NLRB modified the rule to require a notice to the NLRB for the rule to take effect along with a 45-day period after the receipt of notice when a petition could be filed. The notification requirements is what has produced these records, and it’s likely when the requirement is rescinded, the data will cease.

CHIPS

With the assistance of JP Ferguson, we’ve figured out the codes for many of the fields in CHIPS, the 1984-2000ish NLRB case management system.

It’s going to be pretty hard to do much more without more documentation, than what’s available from the National Archive. If you have any CHIPS manuals from the late 1990s or early 2000s, please let me know.

Pre-CHIPS

Also, thanks to JP Ferguson’s blog, I learned that there were a number of electronic records for Unfair Labor Cases, Representation Cases, and Elections on deposit at CISER.

These records go back to the early 1960s and 1970s, which is fabulous.

They are also a bit of a mystery.

The FOIA officer at the NLRB, who is very good and very knowledgeable about the agency’s information systems knows nothing about these data, but the archivist at the Cornell Center of Social Sciences confirms that these records came from the NLRB. If you know anything more about these records, please let me know.

I’m planning on adding these to the the warehouse.

Last month, we welcomed another child into our world, and the parternal urge has been channeled towards refurbishing archives of the two previous NLRB case management systems: the Case Activity Tracking System (CATS) that the NLRB used from around 1999 through 2011 and the Case Handling Information Processing System (CHIPS)–the first NLRB electronic system–that was in operation from 1984 through 2000.

I’ve had versions of these datasets on the labor data warehouse for a while, but I’ve located and added the ULP cases to the CATS data, and converted many columns in the CHIPS data to forms that are much easier to use.

This means that we now have pretty detailed case-level data for unfair labor practice cases going back to 1984. I think this is the first time that this has all been pulled together. We also have the representation cases for the same period, which is also fantastic, but Hank Farber has had that data for a while.

One thing that the CHIPS data helps me see is how interesting the year 2000 is. In his wonderful article, “Union Organizing Decisions in a Deteriorating Environment: The Composition of Representation Elections and the Decline in Turnout”, Hank Farber points out the turn of the millenium as a transition point in how workers vote in NLRB elctions and what types of workplaces organizers bring to the NLRB.

In an analysis that covers some of the same ground, I looked at the changing success rate of NLRB election petitions, and 2000 is the year when the win rate of elections, and petitions really starts to increase.

Farber doesn’t really put forward an explanation about why 2000 was such a pivotal year. His paper has a very nice argument about why we could tend to see a higher NLRB win rate and fewer NLRB petitions (basically, if it’s much more costly for unions to run NLRB election campaigns, they will tend to focus on organzing worksites where they have a high chance of winning). But Farber doesn’t identify anything that was particularly important for making organizing harder in 2000.

I don’t really know either. Maybe you have ideas.

Help with the Unfair Labor Data

The CHIPS data comes from records on deposit with the National Archive. Unfortunately, their deposit doesn’t have all the lookup tables that you would need to make sense of the data. To figure out, say, which union is involved in a representation certification case, we need to link a numeric ID to a table of union names.

The information we need for that is in a big codebook, but it needs to get transcribed into tables. Let me know if that’s something you might want to help with.

FOIA Corner

Two interesting FOIAs came in recently:

• In April 2022, the FMCS started offering card-check services. From April 2022 through November 2022, FMCS has counted cards for 51 groups of workers covering at least 1,544 workers.
• I got back the FOIA from the Illinois Labor Relations Board that I discussed in a previous post. It was a great response and very heartening that this approach could succeed.

Working with the CATS and CHIPS databases, there’s a lot of data that was recorded in those data systems that is not publicly exposed to on the NLRB website. So, I also submitted a FOIA for the database schema of NLRB’s current case management system NxGen.