Abstinence in midlife was associated with a 45% higher risk of dementia compared with people who consumed between one and 14 units of alcohol per week.
However, the underlying mechanisms are likely to be different in the two groups.
Presenting its findings the researchers stated: "Given the number of people living with dementia is expected to triple by 2050 and the absence of a cure, prevention is key". The UK guidelines suggest an alcohol threshold of 14 units/week but many countries use a much higher threshold to define excessive consumption. "We show that both long-term alcohol abstinence and excessive alcohol consumption may increase the risk of dementia".
Researchers say it's almost impossible to definitively determine the effect of alcohol consumption - as it would require a trial in which participants would have to stop drinking or start drinking heavily. Researchers traced participants' health records for dementia using the databases of the national hospital episode statistics, the Mental Health Services Data Set, and the mortality register.
A total of almost 400 dementia cases - with onset occurring, on average, at age 76 - were reported. During this time dementia was diagnosed in 397 participants.
"Future research will need to examine drinking habits across a whole lifetime, and this well help to shed more light on the relationship between alcohol and dementia".
But the risk also increased by 17 percent for every three additional drinks above six.
In the United Kingdom, 14 units of alcohol a week is now the recommended maximum limit for both men and women, but many countries still use a much higher threshold to define harmful drinking.
Chronic heavy drinking has been clearly established as a major risk factor for all types of dementia, especially the early onset of the disease.
The team of French and British researchers suggested part of the excess risk of dementia in abstainers could be due to a greater risk of cardiometabolic disease reported in this group. But the increased risk of disease and those who completely refused alcohol. In moderation, of course, unless you want to increase your chances of developing dementia.
Dr Doug Brown, chief policy and research officer at Alzheimer's Society, said: "By finding evidence that drinking lots of alcohol, and also drinking no alcohol at all both increase dementia risk, this study supports other work that continues to question whether drinking up to the equivalent of six glasses of wine per week might have a protective effect against dementia".
"This study is important since it fills gaps in knowledge, but we should remain cautious and not change current recommendations on alcohol use based exclusively on epidemiological studies", according to the paper.