It looks very similar to the warehouses at Pepper/Woodford, which were built in something like 1880, and I suspect this place is from the same era, so it is almost certainly not "stone from 1840," not that it matters.
Several distilleries used native limestone for some of all of their buildings. Kentucky River (later Canada Dry) did, so did Stamping Ground. Generally, though, stone construction was not used. Kentucky distilleries tended to look rather thrown-together. One writer said they looked like sawmills.