Previous part

In the previous part of this article we've looked in detail at the Swiss system of direct democracy.

It's hard not to conclude that direct democracy is the reason why Swiss society works so well. However, one then remembers the direct democracy in California and is no longer so sure. Not that the Californian direct democracy is necessarily bad, but it shows no spectacular results either.

One possible argument to explain the discrepancy could be that direct democracy needs time to mature. Which it certainly does. We've already seen how the Swiss popular initiative needed hundred years to evolve to its current form. However, Swiss direct democracy dates back to the end of 19th century, while Californian direct democracy became fully established in 1911. Difference of mere 22 years.

Another possible explanation of why Swiss direct democracy works better is that California is constrained by being part of the United States. And there is no direct democracy on the federal level in the US. Maybe the dysfunction trickles down from above and the local direct democratic institutions are not able to cope with it.

I have no idea whether that's a plausible explanation, however, we may be wiser after giving a closer look to the second pillar of the Swiss political system, the feature that is often called "federalism" but which can, in my opinion, be interpreted in much wider sense as decentralization at all levels of the society.

Cantons

Switzerland has once been a loose confederacy of independent states and the unification in 1848, to a significant extent, preserved that independence.

While there is a central government, its powers are rather limited. So-called "principle of subsidiarity" applies: The areas managed by the federal government are explicitly listed in the constitution and anything that is not included in the list automatically falls under the jurisdiction of the cantons. Even in the areas that the federal government is responsible for, it often does the decision making, but leaves the execution to the cantons. So, for example, although the Department of Justice and Police exists at the federal level, the police forces themselves operate at the cantonal level, in larger cities even at the municipal level.

Each canton is a small state on its own. It has its own constitution, its own parliament and its own government.

Even the political system varies among the cantons. Most cantons hold cantonal elections every four years, but the canton of Appenzell Innerrhoden holds them every year. Yet other cantons elect their parliaments every five years. Most cantons use majority system in cantonal elections. However, canton Ticino uses proportional system. In canton Bern, one seat in the cantonal government is reserved for a representative of the French-speaking minority in Bernese Jura. In the canton of Glarus, the voting age was lowered a few years ago and therefore, unlike in the rest of Switzerland, young people can vote from the age of sixteen.

The cantons can negotiate directly with each other and adopt mutual treaties (the so-called "concordats") thus bypassing the federal government. Powerful intercantonal organizations exist: The Conference of Cantonal Governments, the Conference of Cantonal Ministers of Finance, the Conference of Cantonal Ministers of Education and so on.

While these parallel structures make it possible to keep the central government lean, some argue that they are lacking the full democratic accountability. As far as I can say, it doesn't look that bad though. When German-speaking cantons were trying to harmonize their school curricula, the measure was voted on in cantonal referenda in all the affected cantons rather than having it forced on them by the conference of education ministers.

Sometimes it happens that the cantons are unable to agree on a question. One such question was whether the school year should begin in spring or in autumn. That was, understandably, a serious problem for the families moving between the cantons. The issue was resolved in 1985 in a federal referendum (58.8% in favor of autumn). The decision was written into the federal constitution and Switzerland took a small step towards centralization.

Municipalities

The power sharing between cantons and municipalities is governed by the cantonal constitution and thus varies among the cantons. German-speaking cantons tend to apply the subsidiarity principle. Everything that is not explicitly delegated to the canton is governed on the local level. Canton of Graubünden goes the furthest. It used to be, historically, a federation of municipalities, or rather, a union of three federations, called - the Game of Thrones fans are going to appreciate it - The Grey League, The League of Ten Bailiwicks and The League of God's House. (And it gets stranger. What is now the canton of Wallis was once called the Republic of the Seven Tenths.) In any case, in Graubünden the villages have their own constitutions and so on. The French-speaking cantons, on the other hand, are less fond of passing power to the lower levels. The municipalities, for instance, often have no say it their political system, which is, instead, imposed on them by the cantonal constitution.

Either way, municipalities are relatively separate units and there is a strong pressure on them to remain so. This is evidenced, for example, by the fact that all attempts to unify the tax system, whether within a single canton or nationwide, have been rejected in referenda.

Municipalities can enter into mutual agreements (e.g. regarding the joint supply of water), but, interestingly, they cannot negotiate directly with the federal government. The federal government, in turn, has no direct relationship with the municipalities. It must always act through the canton.

An unusual detail is that municipalities are in charge of granting citizenship. Swiss citizenship is acquired by citizens of the individual cantons and cantonal citizenship is acquired by citizens of individual municipalities. The process therefore bubbles up in the bottom-up way.

To understand the split of powers and responsibilities between the federation, the cantons and the municipalities, let's have a look at how are individual sectors financed.

Municipalities spend the most on the cost item "environment" (63%). Environment is followed by "culture, sports and recreation" (56%) and "administration” (44%).

By comparison, the cantons contribute the most to the items "public order" (67%), "health" (56%) and "education" (53%).

Federal government, in turn, spends the most in the area of "foreign relations" (100%), "defense" (90%), "economy" (52%), "transport" (51%) and "finance" (51%).

Similarly, taxes are collected on federal, cantonal and local level. Each level determines the tax rate on its own. The ratio between the three of them tends to be around 1:1:1.

All taxes are collected at the municipal level (although sometimes the municipality can delegate the task, for a fee, to the canton). The municipalities then send the corresponding portion of the taxes to the cantons and these in turn to the federal authorities. The other way round, the central government sends, within the framework of equalization programs, part of the money back to the cantons. The equalization is relatively moderate though and makes up at most 15% of the cantonal income.

Since cantonal and local taxes can account for up to huge chunk of overall taxes, rich people are incentivized to move to tax haven cantons, such as Zug or the predominantly rural canton of Schwyz. This begs a question whether there is a tax race to zero among the cantons and municipalities. The data seems to indicate that indeed, the taxes tend to slowly decrease. But, on the other hand, the public sector doesn't seem to be underfinanced at the present moment.

Associations

Below the municipal level, there are associations of different kinds: Church communities, professional organizations, guilds, clubs for keeping public spaces clean, sport clubs, shooting clubs, gardening clubs, cultural associations, parents' associations, charities, choirs and so on.

Switzerland is home to around 90,000 associations and most Swiss are members of at least one association.

Depending on where you come from, you may feel that putting associations on the ladder of decentralization is strange. Governance, in the end, is something essentially different from hobbyism. In many countries the split between the official (mandatory, political) and the non-official (voluntary, for fun) is quite palpable. But in Switzerland the boundary is quite blurry.

Military service, for example, is mandatory and once a person completes basic training, they get a gun. From that point on, they are required to practice shooting at least once a year. Given that the shooting is mandatory, one would expected it to be organized by the state. In reality, however, it's the local shooting clubs who take care of the task. In other words, the state relies on voluntary associations to perform some of its critical tasks.

Or have a look at so called "citizens' municipalities". These are associations that own the the municipal land, public buildings, forests and mountain pastures. Citizens’ municipality Bern even owns a bank. The "citizenship" in these municipalities is hereditary and distinct from the citizenship in the "political municipalities" familiar from elsewhere. Citizens' municipality collects rent from its possessions and spends it on causes beneficial to the municipality. You are left to wonder whether these are state or private organizations.

Militia System

Another mechanism that blurs the border between the state and the citizens is the so-called "Militia system". It is a principle according to which the civil service is voluntary and performed in one's free time, with no or little compensation.

From that point of view, there's no much difference between being a treasurer of the municipality and being a treasurer of the local yodeling club. Also, both state and non-state organizations are under public scrutiny. Non-functional minicipality requires fixing as much a non-functional rowing club.

Militia system deserves a few comments.

While it is based on the age-old concept of "citizen and soldier" and, indeed, every Swiss male is obliged to do military service, it extends to other spheres of public life. Citizens can be asked, for example, to do service in a local fire brigade. This is generally frowned upon and volunteers are preferred. Nevertheless, the option exists. Voluntary service in school boards, churches and in municipal, cantonal or federal administration is also considered to be a part of the militia system.

In the latter case, the militia system is meant to bring real world, practical experience to the government. The idea is that a baker can represent other bakers in parliament, whether federal or cantonal, better than a professional politician.

However, the militia system seems to be gradually breaking down. Only a third of members of the federal parliament are still doing it as a side job. Also, with the increased mobility, the feeling of belonging to a particular place and the prestige associated with working for a municipal authority wears down. Large municipalities tend to hire salaried employees. Small municipalities, on the other hand, are struggling to fill in the administrative positions. This is one of the reasons behind the increasing number of municipality mergers in the past few decades.

In the following graph, blue is the overall number of municipalities in Switzerland. Red is the average number of inhabitants per municipality.

If you put your conservative hat on, this development is deeply troubling. Maybe even more troubling than the current bogeyman of the progressives, the rise of right-wing populism. Because what it means, on the big scale, is replacement of the state run by citizens by the state run by the political class. In other words, the replacement of participative democracy by a spectator democracy. On the local scale, on the other hand, with at best as many candidates as there are positions the election turns from choosing the best candidate into approving the single running one. And it is not clear, whether the unique Swiss model of direct democracy and rule by consensus can survive that.

There is no easy way out. Even making the administrative service compulsory may not be sufficient. Some cantons do have such a provision in their constitutions, inherited from the times past. But now, that they actually put it in use, it turns out that the enforcement is hard. Couple of years ago, for example, a small village in the canton of Uri has elected a new municipal council. Out of five members, three were elected against their will. To avoid the service, all three have relocated, leaving the council without quorum and therefore non-functional.

Apartment blocks

At the very bottom, at the bottom of the decentralization ladder, there are the politics of apartment blocks. These are things like parking (Your car is not parked in parallel!), noise (The neighbor is playing trumpet at 10pm! And he's butchering the song!) and similar.

This is often very informal and varies from one apartment block to another. I can't speak for everybody, but let me quickly summarize my own experiences as well as the anecdotes I've heard from others.

For me, personally, the most palpable demonstration of apartment block politics is the shared laundry room, which is very common in Switzerland. It is a direct attack on the modern custom of people living side by side without knowing each other or solving common problems.

The shared laundry room was, for example, what led to my very first contact with my neighbors. The very first week, a neighbor complained that I hadn't wiped water from the rubber band around the door of the washing machine and gave me a long lecture about the rules for using the shared washing machine and tumbler.

In theory, everyone is free to buy their own washing machine, but bathrooms are often designed so that there is no space for a washing machine. And it's not a custom in the first place.

Our block consists of six apartments and the washing days are divided between the apartments. Each gets one day of week, from Monday to Saturday.

Problems frequently arise. Six families sharing a single washing machine are bound to clash. One may have returned from a holiday and would like to do their laundry quickly. Some families may have a toddler and therefore more laundry to do. Some of the neighbors may be more willing to share, some of them less. Some people may interpret house rules in different ways. All of that requires constant diplomacy, keeping good relationships and tit-for-tat arrangements.

Sometimes it happens that someone does their laundry on Sunday, which is not supposed to happen. None of the neighbors complains, because everyone is aware that they may be in urgent need of doing laundry on Sunday themselves at some point. At the same time, however, everyone pays attention to whether people are not abusing this freedom and doing laundry on Sunday regularly, without a serious reason.

The point of all this is that many basic virtues of participative democracy, such as a proactive approach, negotiation, willingness to compromise, acceptance of a common decision, and even the ability to turn a blind eye, are drilled into people every day when dealing with such details as the use of the shared laundry room.

Why decentralization?

And so one asks: Why all the decentralization? Why all those local and regional, intertwining rules that complicate the whole system so much that it's barely understandable? What is it all good for, anyway?

Friedrich Hayek is the author of a famous parable about how the market works as a device for collecting and processing information. Everyone is free to decide what to buy and at what price, providing the market with information about their preferences and financial possibilities. Manufacturers, in turn, are free to sell at any price, providing the information about the manufacturing costs. Market takes care of the processing and produces the right amount of each good, at the right time, at the right price. No central planner would be able to measure the tiniest preferences with a similar accuracy and create a plan that would produce and ship the goods exactly to where they are needed.

What's much less known is the work of another Nobel laureate, Elinor Ostrom. What she says is similar to what Hayek says, except it's about institutions, rather than markets: Proper institutions function as information collecting and processing devices. They process the messy hands-on knowledge of the stakeholders and transform it into efficient administrative rules. If the current rules do not work, the participants, being personally involved in the whole matter and knowing all the details, will adjust the rules to solve the problem.

Ostrom identifies several principles that must be followed to make such an institution work. I am not going to list all of them. If you are interested in the topic, read my review of Ostrom's book here. The part that particularly interests me in the context of this article is the rule stating that policies must be fitted to local conditions.

As an example, Ostrom describes a village in Sri Lanka, where fishing was practiced at two different spots. The fishing grounds were about one and a half kilometers apart. Each of them had different rules on who, how and when could fish. These rules have been carefully crafted to take sea currents, changes in the abundance of fish during the day, the cost of preparing the equipment, etc. into account.

The rules we are talking about here are based on many years of experience at the local level, and no central government, no planner, however ingenious, would be able to design a similarly optimal system. The best that could be expected would be a well-thought-out, but rigid system that would apply the same rules to every village on the Sri Lankan coast. That, in turn, would lead to sub-optimal use of resources in some places and, conversely, to over-fishing and gradual destruction of the fisheries in other places.

But to return to Switzerland: Let's take the already mentioned case of the Rosengarten tunnel, a project to divert traffic from the busiest Zurich street to the underground. What I hear from the locals is that many voted against the tunnel, because if it were built and the traffic really went underground, the whole neighborhood would suddenly be more lucrative and the rents would increase. (And again: High rents are a pain point in Zurich.)

Those who rent a flat in the neighborhood therefore have a good reason to vote against the tunnel. However, the situation is the opposite for those who own real estate there. If apartment prices increase, they will benefit from it. The result of the vote is therefore influenced by the ratio of subtenants to apartment owners. No official would be able to take this particular detail, as well as many other, no less important, into account and make a balanced decision.

The same is true when setting up a political system: The system must be adapted to local conditions. Let us remember the one already mentioned in the government of the canton of Bern for the representative of the French minority. Such a rule doesn't make sense elsewhere. Some cantons have no French minority. Others are almost purely French. The rule only makes sense for the canton of Bern, because the French-speaking Bernese Jura has, for historical reasons, a special status. Other cantons it turn need different rules.

Finally, it should be said that the cantons often serve as political laboratories. Rules that have worked well in one canton are sometimes adopted at the federal level. This allows for relatively safe experimentation without dire consequences for the entire country. When, for example, proportional voting system was introduced on the federal level in 1917, it had been already tested in the canton of Ticino. When voting age was lowered to sixteen in the canton of Glarus in 2010, some have expected that the innovation would spread to other cantons, and maybe even to the federal level, counterbalancing the graying of the electorate. However, that has not happened yet.

The Most Powerful Man of Switzerland

When a photograph of the Swiss president waiting for a train trended on Twitter in 2014, people were amazed at what a safe country Switzerland has to be if the president can take a train just as any other mortal. Others flipped over Swiss egalitarianism, over the country where a farmer and a worker could find themselves in a train compartment with the president.

Few have realized that the prosaic explanation of the fact is simply that the president is not important enough to have to be transported in an armored limousine. In fact, many Swiss people have trouble remembering who the president happens to be this year.

So wait, if the president is not the Switzerland's most powerful man, maybe it's the prime minister instead?

But no. Switzerland has no prime minister. The government has seven members, all of them equal, deciding on issues by voting.

The federal chancellor exits the competition straight away. He's nothing like Angela Merkel. He plays a technical role and does not even have a vote in the government.

The powers of the federal government are very limited anyway. Most of the power remains at the cantonal level. So perhaps the most powerful woman is the president of the most populous canton, which is, with its million and half inhabitants, Zurich.

But what was already said about federal government, applies to the cantonal government as well. The position of cantonal president rotates annually among the seven members of the cantonal government, and the cantonal constitution gives her almost no special powers. Section 23 of the cantonal constitution goes the furthest: "The cantonal government may instruct the president or vice president to decide on matters of minor importance."

It turns out that Switzerland is not ruled by anyone in particular. And that brings us back to the subject of decentralization. The lesson we can learn here is that decentralization does not necessarily mean only that some powers are transferred from the state to the region, or perhaps to the municipality. Decentralization, in a broad sense, is a way of political thinking that seeks to prevent accumulation of power. And it does not matter whether it is an accumulation of power in the hands of one person (president), in one institution (government), or in one place (capital).

Power is hopelessly diluted. Nobody can make a decision on their own. Everyone is forced to negotiate and, eventually, to compromise. This arrangement is one of the contributing factors to the Swiss system of ruling by consensus, the so called "concordance democracy", which we are going to explore in the next installment of this essay.

July 22nd, 2020

by martin_sustrik

Swiss political system may be best known for its extensive use of referenda. However, others may argue that its most striking feature is the ability to avoid political polarization. In this respect it may be unique among the western nations.

That being said, it is hard to learn much about how it works. First, a big part of the system is informal and thus only discoverable by observing it personally or by asking the locals. Second, it's strongly decentralized. Different rules apply in different cantons and municipalities which makes the topic confusing to study. Third, Swiss aren't especially interested in promoting their own system abroad. A lot of the resources therefore exist only in local languages.

In this article I'll try to put together what I've learned by living in the country, speaking to local people, following local press and studying the resources.

Still, a disclaimer is due: I am not Swiss. I have lived here only for five years. Neither am I a political scientists or a sociologist. If you are Swiss, or simply know better than me, let me know about any inaccuracies in the article.

On the more technical side of things: There's a lot of material to cover, and the result may be rather overwhelming. It would be a small book rather than a long article. Therefore, I am going to split this essay into three or four installments which I will publish one at a time.

Semi-direct Democracy

When modern Switzerland was established in 1848, it was a pretty standard representative democracy, mostly based on the American model.

It's a federal state. Federal elections are held every four years. People are represented by political parties. There are two chambers of the parliament. Parliament elects members of the government, who then together run the country. The thriving ecosystem of various voluntary associations resembles the America that Alexis de Toqueville has written about.

However, Switzerland is special in that various elements of direct democracy were introduced in the course of history.

There are obligatory referenda: Any change in constitution, adjustment of taxes or joining any international organization must be approved by the people and the cantons. There are legislative referenda: Any law enacted by the parliament may be challenged and rejected in a referendum. Finally, there are so called "popular initiatives" which can propose a referendum on any topic. If the initiative manages to collect specified amount of signatures within specified amount of time the referendum is organized and the initiative may eventually get enacted. All of these referenda exist not only on the federal, but also on the cantonal and the municipal level. All of them are binding and neither of them needs a quorum.

To understand the scope of the thing, consider that a 36-year-old from the city of Zurich who turned 18 in year 2000, has, in past 20 years, had the opportunity to take part in 548 referenda, 181 of them being on the federal, 176 on the cantonal and 191 on the municipal level. With the average turnout of 45% it means that they have voted in approximately 246 referenda.

Due to their large number, individual referenda are not organized separately. Instead, they are voted on in batches, typically four times a year.

To get a flavor of how it feels like, here's the batch from the city of Zurich in February 2020:

• popular Initiative "Affordable Housing": A sensitive issue especially in big cities like Zurich or Geneva, where rents are some of the most expensive in the world. The initiative proposes to build at least 10% of affordable, non-profit or cooperative flats, as well as a pre-emptive right for cantons and municipalities to buy land. It also proposes that infrastructure upgrades should be done without reducing the number of available flats. The referendum is held at the federal level. 46.5% in favor. Rejected.

• Prohibition of discrimination on grounds of sexual orientation: Switzerland has previously prohibited discrimination on grounds of race, religion, age or political affiliation. This proposal adds sexual orientation to the list. Federal referendum. 63.52% of in favor. Enacted.

• Law on passenger transport in taxis and limousines: A law that introduces the same rules for Uber and similar services and for the traditional taxi services. At the same time, it moves the enforcement of these rules from municipalities to the canton. The law was issued by the government of the canton of Zurich and challenged by a public initiative. (Not the least argument being that the law gives too much power to the canton at the expense of the municipalities.) Cantonal referendum. 52.84% of in favor. Approved.

• Rosengarten tunnel and tram project: A plan by the canton to put 1.1 billion francs into rebuilding the busiest street in Zurich and moving the traffic underground. The plan was challenged by a public initiative. Cantonal referendum. 36.32% of in favor. Rejected.

• People 's initiative "Reduce the tax burden for lower and middle income people": An attempt to reduce income inequality. The proposal adjusts the cantonal taxes by raising the threshold for non-taxable income, as well as by increasing the tax burden in the highest income brackets. Cantonal referendum. 42.04% in favor. Rejected.

• popular Initiative "Lower Taxes for Everyone": A proposal to reduce cantonal taxes for the highest income groups. The aim is to prevent the relocation of the wealthy people to tax havens such as the cantons of Zug or Schwyz. Cantonal referendum. 29.63% in favor. Rejected.

• Partial replacement of the tram depot in Hard district by new communal flats. The city proposes to take a loan of 203 million francs. Municipal referendum. 70.9% in favor. Adopted.

The canton publishes a handbook for each ballot, which explains, in quite a detail, including graphs, maps and tables, what each referendum is about. Take the Rosengarten tunnel project. The guide devotes eight pages to explain the project, including topics such as the impact on the traffic situation in the canton, the impact on the environment, or a detailed explanation of the financing of the project. It states that both the cantonal government and parliament recommend voting in favor of the proposal. It is followed by the opinion of the minority in the cantonal parliament, arguing that the costs are too high, that the financial contribution from the federal government is uncertain, and that the project doesn't really address the existing problem. They recommend to vote against. The next page contains the opinion of the parliament of the city of Zurich. They argue, in rather strong terms, against the project. Finally, there's the opinion of the referendum commission, which is, as one would expect, against the tunnel.

If even the election guide is not enough, you can have a look at the websites advocating for the yes and no vote, respectively. While the website against is relatively minimalist, the in favor side has a long list of supporters. In addition to almost all political parties, there's a long list of supportive associations: The Automobile Club, the Association for the Promotion of Public Transport, the Employers' Association, the Association of Construction Companies of Canton Schaffhausen, the Association of Small and Medium-sized Enterprises, the Property Owners' Association, Swiss Travel Club, Zurich Chamber of Commerce and the like. Many of these organizations have also published their own assessment of the project.

As can be seen, the voters aren't exposed to a simple, black and white choice. Instead, they are drawn into a complex network of different preferences: Your party is in favor, but the deputies of your municipality are against. You are a member of the automobile club and the club is in favor. But your neighbors are against. Voting necessarily means understanding that things are never clear-cut.

Mandatory Referenda

Any change to the constitution must be approved by the voters in a referendum. There's no way around it. If you want to change the constitution, you need the majority of voters and the majority of cantons to vote for it. Period. (To clarify: Canton is considered to be in favor if the majority of voters in the canton are in favor.)

While this may seem as a reasonable rule on its own, it is in fact an important piece that complements the overall system. The results of popular initiatives are, for example, written into the constitution, meaning that they can't be overturned, except by a different referendum. (On the other hand, it gives the Swiss Constitution a rather special character. It begins with the thundering: "In the name of Almighty God! We, the Swiss people and cantons, mindful of our responsibility to the Creation" etc., but then it ends with guidelines for the protection of swamps and rules for building holiday homes.)

Similarly, Switzerland has no constitutional court. The right to interpret the constitution is granted only to the people. They may do so by running a referendum that makes the wording of the constitution more clear.

In short, the system is crafted in such a way that there are no loopholes. No way to disrespect the popular opinion.

In addition to the changes in constitution, referendum is also required to to join international organizations. This way, Switzerland decided not to enter the European Economic Area in 1992, to join Schengen area in 2005, not to join UN in 1986 and, again, to join UN in 2002. (And yes: Palace of Nations, the headquarters of UN, is located in Geneva and was located there for a long time even before Switzerland has become a member.)

Legislative Referenda

Legislative referenda get the least publicity but they may be the most important of all. Unlike constitutional referenda and public initiatives that tend to focus on big topics the legislative referendum can challenge and reject any law, no matter how trivial, passed by the parliament.

This keeps the parliament and the government in check on day-to-day basis. To quote Wikipedia:

The possibility for the citizens to challenge any law influences the whole political system. It encourages parties to form coalition governments, to minimize the risk that an important party tries to block the action of the government by systematically launching referendums. It gives legitimacy to political decisions. It forces the authorities to listen to all sectors of the population, to minimize the risk that they reject new laws in referendums. Before presenting a new bill to the parliament, the federal government usually makes a wide consultation to ensure that no significant group is frontally opposed to it, and willing to launch a referendum.

In short, legislative referenda are probably the single most important force that driving Switzerland away from the political polarization and towards the rule by consensus.

Popular Initiatives

Popular initiative is a way to partially change the constitution in arbitrary way.

As has already been said, if any proposal collects hundred thousand signatures in a year and a half, it is voted upon in a referendum. The result of the vote is binding and there is no quorum. If just 1% of the population takes part and 0.51% votes in favor of the proposal, it will be enacted and implemented.

Also, there are no restrictions on the topic of the popular initiative. In some countries that have similar instrument in their constitution the topics are restricted. It may not be possible to hold referenda about basic human rights or maybe about taxes. Not so in Switzerland.

To understand what a popular initiative means, let's have a look at a little sample. What follows are all the popular initiatives on the federal level that were voted on in the 2015-2019 election period:

• "Stop urban spread." The Young Greens' initiative against suburbanization and for stricter zoning. 36.3% in favor. Rejected.

• "Swiss law instead of foreign law." Proposal for the Swiss constitution to take precedence over international treaties. Referendum initiated by the Swiss People's Party. 33.7% in favor. Rejected.

• "For cows with horns." The initiative initiated by farmer Armin Capaul. It proposes to subsidize the farmers who do not cut the cows' horns. 45.3% in favor. Rejected.

• "For food independence." A complex proposal to support farmers. It includes a ban on genetically modified organisms. 31.6% in favor. Rejected.

• "Fair-food initiative." The Greens' attempt to introduce restrictions that would promote fair, environmentally friendly agriculture and prevent food waste. 38.7% in favor. Rejected.

• "For full-reserve banking." The initiative proposes that the Swiss National Bank should be the only source of money. Other banks would have to have cash reserves sufficient to pay out all the deposits. Initiative of the association "For the Modernization of Currency". 24.3% in favor. Rejected.

• "Against radio and television fees." Publicist Olivier Kessler's proposal to abolish fees for state-owned media. 28.4% in favor. Rejected.

• "For phasing out nuclear energy." An initiative launched by the Green Party. It proposes to decommission all the Swiss nuclear power plants by 2029. 45.8% in favor. Rejected.

• "For strong social insurance" An initiative of the largest Swiss trade union. It demands to increase payments to the social insurance by 10%. 40.6% in favor. Rejected.

• "For green economy." The initiative calls for the Swiss economy to function in a sustainable way. The government should set goals and report on how they are achieved at each session of the parliament. If the progress lags behind, additional measures should be taken. 36.6% in favor. Rejected.

• "For universal basic income." An initiative was initiated by several individuals. It proposes an unconditional regular income for all. The amount of income and the method of financing should be determined by law. 23.1% in favor. Rejected.

• The so-called "Dairy Cow" initiative. It suggests that the entirety of the fuel tax income should be spent on road maintenance. 29.2% in favor. Rejected.

• "Pro Public Service." The constitution should explicitly stipulate that state and semi-state organizations (post office, railways, telephone) are not run for financial gain. It also limits the salaries of the employees in these organizations. 32.4% in favor. Rejected.

• "Stop food speculation!" Young Socialists' initiative. It proposes to ban certain financial instruments in the area of ​​agricultural products. It orders the Federal Government to combat such practices also at the international level. 40.1% in favor. Rejected.

• "For enforcing the expulsion of criminal aliens." Initiative of the Swiss People's Party. The party was dissatisfied with the government's implementation of the successful referendum on the expulsion of criminal aliens held in 2010. 41.1% in favor. Rejected.

• "Against fines for marriage." The initiative of Christian Democrats, who did not like that in some cases unmarried couples paid less taxes than married couples. 49.2% in favor. Rejected.

Small cantonal or municipal popular initiatives are probably not that interesting for a reader from abroad, but still, let's mention a few of them. In recent years, the voters in the canton of Zurich have voted on: Definition of marriage as a union of a man and a woman. For the expansion of the Stadelhofen railway station. For the replacement of hunting associations by professional nature conservationists. For the harmonization of school curricula in German-speaking cantons. For one, instead of the two, compulsory foreign languages ​​in schools. For the law to support for the film and gaming industry. For economic organizations to take part in funding kindergartens. For the abolition of the commission reviewing the claims of rejected asylum seekers. For effective control of minimum wages. All of those initiatives were rejected.

It is also worth looking at the history of popular initiatives.

When modern Switzerland was founded in 1848 there was a clause in the constitution that the people could change the constitution. It was generally interpreted to mean that the constitution could only be replaced in its entirety. The instrument of the popular initiative was not established until 1891.

When we look at the list of all the popular initiatives, we notice that the instrument of popular initiative was little used at the beginning. The number of popular initiatives soars only in the seventies. The graph shows the number of all popular initiatives in blue and the number of successful ones in red.

The reason is that only by then did the instrument get working really smoothly.

In the beginning, for example, the custom of "putting the initiatives into the drawer" has become established. The new initiatives were simply left in the vacuum, without a referendum, until they were forgotten, or until the initiative had lost all of its political relevance. One particular initiative was literally forgotten and canceled only after spending 43 years in a drawer.

After this system was heavily criticized in the press, the government eventually gave up on it.

The next trick was to make a government counter-proposal for a popular initiative and thus divide its supporters. If, say, 60% of the people were in favor of the initiative, the two proposals (the original proposal and the government's counter-proposal) divided them into two groups of 30% each, so that neither proposal passed.

This problem was solved in 1987 by the introducing so-called "double yes" which makes it is possible to vote for both the initiative and the official counterproposal. An additional question has also been introduced which asks which of the proposals one would favor if both proposals were successful.

Next, there is the problem of the validity of the referendum.

The Swiss constitution does not limit the subject of the popular initiative in any way. The only requirement it makes is that it has a coherent content. In practice, this means that the voter should never be forced to say yes or no to a question that mixes two unrelated matters. (Example: Do you want Putin to be able to run for a president for two more election periods and adjust the state pension in line with the inflation?)

So, for example, the popular initiative which called for a reduction in military spending and the use of the money for social purposes, was canceled. The government's argument was that the financing of the army and the financing of social affairs are two independent issues that cannot be conflated in one referendum.

The argument sounds reasonable. But then one notices that some of the constitutional changes initiated by parliament are cheerfully mixing changes in various parts of the constitution. The system is unbalanced in this respect and the problem has not been solved yet.

Next, there is the problem of the consistency of the proposal with international treaties.

The first historical case had to do with a contract with Germany about the construction of a hydroelectric power plant in Rheinau on the border of the two states. According to the contract, the concession could not be canceled unilaterally. When the government, in 1954, allowed the popular initiative for the abolition of the power plant, it opened up the question of what happens if a referendum contradicts Switzerland's international commitments.

Back in the day, Switzerland avoided embarrassment because the initiative against the Rheinau power plant has not been successful. In recent years, however, there have been couple of successful initiatives that contradict international treaties.

One of them was the initiative for the automatic expulsion of criminal aliens in 2010. As the result of the referendum was never properly put into practice, in 2018 the original author of the initiative (Swiss People's Party) comes up with a different initiative proposing that the Swiss constitution - and therefore the results of the popular initiatives - should always take precedence to the international law - with the exception of those international treaties that were approved in a referendum.

Should a referendum pass, Switzerland could at any time revoke its existing international obligations and would be considered an unreliable partner abroad. Which, of course, could be a serious problem for Switzerland's export-oriented economy.

However, the referendum did not pass and so the problem is still unresolved. We can only guess how it will turn out. Maybe, one day, all international treaties will be voted on to gain unquestionable legitimacy. However, even this will not solve the problem of already existing international treaties and retroactive changes through popular initiatives.

To explore another serious and hard-to-fix flaw in the Swiss political system, let's have a look at the initiative "Against Mass Immigration".

First, some background.

Immigration is a serious issue in Switzerland. In that it differs from certain countries, including my native Slovakia, where immigration is negligible, but it is nevertheless used as a bogeyman to score political points. In Switzerland, a quarter of the country's population does not have Swiss citizenship. In city cantons such as Zurich or Geneva, the proportion of foreigners is even higher. There's even a lot of third generation immigrants who still don't hold Swiss passport.

The problem began after the second world war, when Switzerland, spared by the war, became an attractive country to immigrate to. People started moving in and that caused political tensions, as witnessed by no less than seven referenda against immigration between years 1968 and 2000. As Max Frisch once pointedly noted: "We asked for workforce and people came instead."

The number of non-citizens is nowadays so high that it's not only the xenophobes who lose their sleep. Traditional conservatives are worried as well: Is it possible to preserve the existing communal and political culture with that many foreigners? And so are liberals: Can a country where quarter of population doesn't have the right to vote be still called a democracy?

After all the anti-immigration referenda failed during the second half of the 20th century (support varied from 29.5% to 46.3%), initiative "Against Mass Immigration" finally succeeds in 2014 with 50.33% of the vote in favor. It asks for introducing quotas for foreigners, such that they "align with Switzerland's economic interests and favor Swiss citizens."

The government announces that it will act quickly and pass the necessary legislation before the end of the year. One week after the referendum, Swiss Minister of Justice calls Croatian Minister of Foreign Affairs and informs her that Switzerland won't sign the draft agreement, which gives Croatia (then a new EU member state) free access to the Swiss labor market.

The European Commission responds that one can’t cherry-pick from the freedoms enshrined in the treaty and that restricting freedom of movement will jeopardize Swiss access to the single European market. Brussels promptly suspends talks on cooperation in the sphere of education (Erasmus+ project with the budget of € 14.7 billion for the next six years) and science (Horizon 2020, € 80 billion budget for the same period). It also suspends talks on integrating the Swiss electricity market into the European market.

Universities report estimated losses on research grants in order of hundred millions euros. The government itself estimates that exclusion from Horizon 2020 will jeopardize 8,000 jobs. Higher electricity prices are expected. The student union is protesting because students suddenly do not know if they will be able to start the planned student exchanges. Credit Suisse is lowering its estimate of Swiss economic growth (from 1.9% to 1.6%) and expects that about 80,000 fewer jobs will be created.

The government finds itself between a rock and a hard place. After three years, just a few months before the deadline for implementing the referendum expires, it abandons the idea of ​​immigration quotas and introduces few half-hearted bureaucratic obstacles to employing EU citizens.

The country suddenly finds itself in an uncomfortable situation where the law is, strictly speaking, in a conflict with the constitution. (Recall that the results of referenda are written into the constitution.) At the same time, Switzerland does not have a constitutional court, which would reject the offending laws. Only the people are supposed to interpret the constitution through a referendum.

To appreciate how dangerous that is, consider that the Swiss system of direct democracy is based on the people modifying the constitution and government subsequently implementing those changes in law. If government starts to disregard the constitution, the system collapses. People may vote for whatever they want and it would have no effect. They can, in theory, challenge unconstitutional laws in legislative referenda, but interpreting hundreds of pages of legalese is probably too much to ask from an ordinary citizen. At the same time, there's no legal instrument to either challenge a standing law after it’s in place for 100 days or to introduce a new law by means of a referendum.

To be fair, a new form of initiative, so called “general popular initiative” was introduced by the government in 2003 that allowed for changing federal law. The instrument was approved in a referendum (70.3% in favor) but later it turned out that a lot of voters had no idea what it was about. In fact, it turned out that even government haven’t had a good idea. When they tried to implement it they found out that there are so many pitfalls and complications that it’s not feasible. In 2008 they proposed that the new instrument is removed from the constitution and the people approved the removal in a referendum (67.9% in favor).

In any case, the story continues with People's party, the initiator of the anti-immigration referendum shouting treason, but then announcing that they will not challenge the decision in another referendum. Instead, they opt to challenge the outcome indirectly, with an initiative asking for Swiss law - and thus the outcome of the anti-mass immigration initiative - to take precedence over international treaties - and thus over the Treaty on Free Movement with the EU. The initiative is rejected in a referendum.

In 2018, signature collection begins for a new referendum. The proposal instructs the government to negotiate the removal of free movement clauses from the treaties with the European Union and, if that does not happen, establishes automatic termination of said treaties. The referendum was scheduled for May 2020, but postponed due to the coronavirus epidemic. That being said, the surveys show that the referendum will most likely fail and the problem of discrepancy between the constitution and the law will persist.

To conclude, it is worth noting how the discussion is becoming more and more nuanced over the years. In the 19th century, it was disputed whether a partial change in the constitution through a popular initiative was permissible at all. Then we see the government openly sabotaging the legal instrument. Today, 130 years after it was introduced, the Swiss are finally dealing with the actual messy problems that the usage of popular initiatives entails.

Dangerous Referenda

A common argument against referenda is that they are dangerous. Let's recall how Lukashenko entrenched himself in the power: The referendum in 1995 gave him the power to dissolve the parliament. In 1996, again in a referendum, Belarusians decided that the presidential decrees would have force of law. Finally, the referendum in 2004 extended the presidential term indefinitely.

Or, for that matter, recall the Brexit referendum and the political chaos it plunged the UK into.

Given this danger and the fact - quite noticeable in the previous sample - that 90% of the popular initiatives tend to be rejected, one has to ask whether Switzerland gains any benefits from using the instrument at all.

But contemplating it, one may wonder whether the fact that out of all 210 popular initiatives that were voted on since 1891 only 22 were successful isn't besides the point. Perhaps it doesn't matter how many initiatives are being rejected. Perhaps the only thing that matters is that there's a safeguard when a conflict of interest between the people and their elected representatives arises. In such a situation a popular initiative may adjust the political system in such a way as to align the interests of the representatives with the interests of the people anew.

But as we look at the successful popular initiatives, we see almost no cases of such initiatives. At least at first glance, it is not clear why the topic of swamp protection or the topic of the construction of holiday homes should lead to a conflict of this kind.

However, two historical initiatives are an exception to the rule.

Back in 1917 Switzerland used to use majority system in the parliamentary elections. This led to a situation where the Liberal Democrats got only 40.8% of the vote, but 54.5% of the seats in parliament. The absolute majority allowed them to pass the laws, regardless of the will of the 59.2% who voted for other parties.

Needless to say, Liberal Democrats torpedoed every attempt to replace the majority voting system by a proportional one. If the instrument of popular initiatives was not available, it would be a dead end. The voters would have to wait until Liberal Democrats lose some of their voter support. But even then, thanks to the majority system, an absolute majority in parliament could be won by another party, who would again find it difficult to abolish the system that brought it to power.

General dissatisfaction with the state of affairs led to the launch of the popular initiative "For a proportional system of elections to the National Council" in 1918 which succeeded with 66.8% votes in favor.

In 1919, elections were finally held using the new, proportional system and Liberal Democrats lost the absolute majority.

The second exception happened in the period after World War II. During the war, a state of emergency was declared, in which a large number of otherwise decentralized powers were transferred to the federal government. After the war, the government refused to relinquish those powers. In 1949, however, the popular initiative "For a Return to Direct Democracy" (50.7% in favor) returned matters to pre-war state.

But however important those two exceptions may be, do they justify the existence of popular initiatives? To justify a powerful and dangerous instrument like that, one would expect to gain at least some day-to-day advantage rather than something that happens twice in a century.

Well, it turns out that referenda, in fact, serve an important day-to-day purpose: They act as a sword hanging over the parliament and the government.

Consider the legislative referendum. It can be used to block any law passed by parliament. The consequence, which, while obvious, does not occur immediately, is that the parliament simply does not pass laws that are apparently going to be challenged and rejected in a referendum.

Often, even a threat of referendum is enough to cause a change in the law or even let it be dropped altogether.

What's more, both the government and the parliament are very well aware of the possibility of a referendum and so they proactively make sure that no significant group of the population has a reason to block the new law.

Additionally, a zero quorum for a referendum means that even small minorities must be taken into account: If the law discriminates against, say, the hearing impaired, the rest of the nation may well ignore the referendum, but the deaf and deaf-mute will still be able to force its abolition.

The popular initiatives complement the system: Legislative referenda can be used only to reject new laws. Popular initiatives can be used to challenge old and dysfunctional ones.

And now it's becoming clear why almost all popular initiatives are rejected. If the initiative had a obvious chance of being approved, the parliament would introduce the necessary legislation on its own. From this point of view the small number of successful initiatives is not a sign of a system malfunction, but rather a proof that the system is functioning the way it is expected to.

In some cases in happens, that initiative has a chance to pass, but the government or parliament considers it harmful or disadvantageous. In these cases, they can come up with a so-called counter-proposal. The counter-proposal is typically a compromise. If the initiative asks for 100, the counter-proposal offers 50. Voters can then choose between rejecting the initiative altogether, accepting it, or accepting the counter-proposal.

In 2014, for example, the Evangelical Party of canton Zurich initiates a popular initiative for reducing the size of classes in schools. The initiative proposes a cap of 20 students per class. The cantonal parliament offers a counter-proposal in which it promises to create 100 new teaching jobs and distribute those teachers preferentially into municipalities that suffer the most from the problems with large class sizes. In the end, voters opted for the government counter-proposal.

The efficiency of the system of counter-proposals is witnessed not only by them being accepted on quite regular basis, but also by the fact that 73 federal popular initiatives were, in the course of history, withdrawn by their initiators in favor of the government counter-proposals.

There are yet more functions of popular initiatives.

To understand the next one, consider the process that each initiative goes through: The Federal Council will first check the referendum and translate it into all official languages. Then, the signatures are collected. The limit for collecting a sufficient number of signatures is one year and a half. The signature sheets are then handed over to the Federal Council. The government has a year and a half to discuss the proposal. If it decides to file a counter-proposal, this period is extended by another year and a half. Consultations with experts and all the stakeholders are held within this time. The government prepares a detailed report and passes it to the parliament. Parliament has another year and a half to discuss it. In the case of a counter-proposal, the period can be prolonged to three and a half years. Finally, the government sets a date for the referendum, which must happen within the next ten months.

The whole initiative, from the draft proposal to the vote, can therefore take up to nine years. In practice, this period usually ranges from two to six years.

The process seems highly inefficient at a first glance, but when one listens to what Swiss political scientists have to say on the matter, it becomes clear that this sluggishness is not a bug, but rather a feature. Some even distinguish between a real referendum (in Switzerland) and a plebiscite (everywhere else). One important difference is that the long duration of the process, which spans across election periods, prevents the referendum from being used for tactical purposes. Another difference is that it provides ample time for in-depth public debate.

And there's a lot of debate. It is not just the consultations organized by the government and parliament. The referenda are discussed in the media, both in serious newspapers and in tabloids, which are handed out for free at tram stops. They are discussed among colleagues at work during the lunch. They are discussed within the family during the dinner. At night they are discussed in pubs and bars. Associations, companies, political parties, government, parliament, all kinds of organizations and individuals all recommend voting either in favor or against. Public discussions are organized. Every simpleton feels obligated to express himself on the subject.

When election day comes, one may get an election handbook that presents both sides of the argument, but at that moment, one's head is already filled of various arguments, both in favor and against. One has become, at least to some extent, an expert on the subject. (And if you think about it, the 548 referenda in Zurich in past 20 years mean that the educational aspect of the system may be surprisingly large.)

To put the above in different words, a popular initiative can also be understood as a call for a public debate on a certain topic. The fact that it is followed by a binding vote ensures that people actually do care about the debate. True, the vast majority of popular initiatives are rejected, but at that point there has been a public discourse and people are at least aware of the matter. With referenda on matters such as universal basic income or full reserve banking, one would expect widening of the Overton window. However, I wasn't able to find a study comparing the size of the window in Switzerland and elsewhere.

One can also think of the public debate as a safety measure. Particular initiative may be dangerous, if approved, but when people go to ballots they are already well aware of the danger.

Another safety measure is that Swiss referenda are, in their essence, not polarizing. In referendum you are never asked to decide between two extremes, between, say, pro-life and pro-choice, but rather between the initiative proposal and the status quo. Voting against is always a safe and neutral option. It doesn't necessarily mean that you are not sympathetic to the spirit of the initiative. You may just think it's going too far, or maybe you like some aspects of it but don't like some other.

Consider the 2013 vote on the law granting special powers to the government in case of epidemic. Some people were against the proposal because they though it makes the federal government too powerful. At the same time they've kept quiet about it because they haven't wanted to be seen as part of the anti-vaxxer crowd which was dominating the debate. Luckily, voting against was a neutral choice they could take advantage of. It haven't meant that vaccination programmes would be relaxed. It just meant that status quo would be preserved.

Referenda as Tools: The Jurassic Question

The history of the Jurassic question begins after the Napoleonic Wars, in 1815, when the Jura region, traditionally part of the Principality-Bishopric of Basel, was annexed to the canton of Bern.

Jura, however, unlike Bern, is French-speaking and to make the situation worse, while the southern part of Jura is predominantly Protestant, same as Bern, the northern part of it is Catholic.

So, starting in 1826, several separatist movements emerge in Jura, fueled mainly by religious frictions, the question of the separation of church and state, and later, to some extent, the nationalism based on the language.

Modern Jurassic separatism dates back to 1947, when the Bern cantonal parliament refused to grant the position of construction minister to Jurassic politician Georges Moeckli on the grounds that he doesn't speak Bernese dialect good enough. That has opened the old wounds.

The following events are chaotic. The emergence of different opposition movements, mutual insults, demonstrations, public burning of a civil defense handbook, demolition of a statue of an unknown soldier, occupation of Swiss embassies abroad, bombs, paving stones and, unfortunately, several casualties.

In short, the whole range of events that accompany separatist movements around the world.

However, unlike in Northern Ireland, where the violence spiraled out of control at approximately the same time, Switzerland has succeeded - not least through the extensive use of the instruments of direct democracy - to keep the situation under control and eventually, if at the typical Swiss sluggish pace, to resolve it.

We can't go into details here, but let's at least look at a short timeline:

• 1968: Establishment of two commissions (one bilateral and one impartial) to propose a plan to address the Jurassic question.
• March 1970: The plan is approved in a referendum. The following referendums are proceeding according to the approved plan.
• June 1974: Referendum on whether to create a new canton Jura. Approved.
• March 1975: Districts that voted against the new canton in a previous referendum decide about their fate. The southern, Protestant part of the Jura decides to remain in the canton of Bern.
• Autumn 1975: Municipalities at the border between the two cantons decide in referenda whether to join Bern or Jura.
• September 1978: In a federal referendum, the Swiss constitution is amended to list the new canton (82.3% of the vote in favor).

Note the architectural beauty of the process. How the referendum is cleverly used to relieve the tension. Step by step, in cold blood, room for manoeuvre is taken from those who benefit from inciting conflict.

Firstly, the referendum on the process of resolving the issue was separated from the referendum on the issue itself. The fact that the process was approved in advance in a referendum gave legitimacy to the following referendums on specific issues and, the other way round, deprived the subsequent attempts to challenge the results of legitimacy.

Secondly, the fact that the process proposed by the preparatory commission had to be subsequently approved in a referendum created pressure in the commission to find a compromise solution. If they leaned too far to one side, there was a risk that the process would be rejected in the referendum and that the entire work of the commission would end up in the trash, along with the political careers of everyone involved.

Thirdly, note how, in the sequence of referendums, it were only those territorial units that voted against the winning solution, that got an additional vote. That prevented unending oscillation between Bern and Jura. The number of disputed areas kept constantly decreasing with each subsequent step of the process.

Finally, the ongoing process siphoned the moderates, who would otherwise have no option but to join radicals, towards peaceful campaigning for the oncoming referenda.

The events do not end with the creation of the new canton in 1979 though.

In the referenda above, the municipality of Laufen decided to remain in the canton of Bern, creating a Bern enclave between the cantons of Jura, Solothurn and Basel-Country. The events continued as follows:

• November 1977: popular initiative "Do you want to start the process of connecting Laufen to the neighboring canton?" succeeds with 65% of the votes for.
• January 1980: Referendum precludes Laufen joining the canton of Basel-City.
• March 1980: In yet another referendum, Laufen decides to start negotiations with the canton of Basel-Country (64.65% in favor).
• September 1983: Unsuccessful referendum on joining the canton of Basel-Country. 56.68% vote against. Laufen remains in the canton of Bern. (A parallel referendum in the canton of Basel-Country approves the adoption of Laufen by a majority of 73% of the votes.)
• 1985: A scandal with discovery of the secret fund to finance Bernese loyalists in Laufen. The Bern Parliament rejects the complaint of the citizens of Lausanne. They bring the case before the Federal Court. The court orders a new referendum.
• November 1989: Laufen decides to join the canton of Basel-Country (51.72% in favor).
• September 1991: Canton of Basel-Country votes to accept Laufen. The decision is less warm than in 1983, but the referendum still passes (59.3% in favor).
• September 1993: Federal referendum approves the annexation of Laufen to the canton of Basel-Country (75.2% in favor).

But the question of the so-called Bernese Jura (Protestant parts of Jura that have not joined the new canton) is still not resolved to the general satisfaction. Separatist haven't yet given up.

In February 2012 the governments of the cantons of Bern and Jura agree to deliver a solution to the problem. In November 2013, two referenda are held, one in Jura, the other in Bernese Jura. The referenda pose the question of whether to begin the process of creating a new canton that would include both areas. Should the referendum pass, a commission would be set up to propose a detailed process, which would then be voted upon in a referendum. The preliminary idea was that every municipality in the Bernese Jura would vote on whether to stay in the canton of Bern or join the canton of Jura.

Although the referendum succeeds in the canton of Jura (64.2% in favor), it fails in Berenese Jura (28.15% in favor). Thus, the question of the Great Jura is definitely off the table. Any further inciting of the Jurassic question loses political legitimacy.

The last painful spot is the town of Moutier, the only district in Bernese Jura which voted for the creation of the Great Jura (55% in favor).

Shortly after the previous referendum, the city of Moutier decides to hold a municipal referendum on joining the canton of Jura.

• January 2016: The canton of Bern approves the referendum.
• June 2017: The referendum accepts the joining of the canton of Jura (51.72% of the vote in favor).
• November 2018: The prefecture of Bernese Jura, complains about the irregularities in the referendum and declares the result invalid.
• October 2019: After the Bernese court confirmed the abolition of the referendum, the city council decides not to pursue the matter in front of the federal court, but rather to hold a new referendum in 2020.

And so, if everything goes well, the Jurassic question will be definitively resolved soon, after more than two centuries of conflict.

July 18th, 2020

Next part

by martin_sustrik

Consider this toy model:

An institution, such as a firm, an association or a state, is formed.

It works well in the beginning. It encounters different problems and solves them the best it can.

At some point though a small problem arises that happens to be a suboptimal Nash equilibrium: Non of the stakeholders can do better by trying to solve it on their own. Such problems are, almost by definition, unsolvable.

Thus the problem persists. It's an annoyance, but it's not a big deal. The institution is still working well and you definitely don't want to get rid of it just because it's not perfect.

As the time goes on, such problems accumulate. They also tend to have unpleasant consequences: If such a problem makes particular medical treatment unavailable, it incentivizes the patients to bribe the doctors and the doctors to break the law and administer the treatment anyway. Now, in addition to malfunctioning medical system, you have a problem with corruption.

After on time the institution accumulates so many suboptimal Nash equilibria that it barely works at all.

The traditional solution to this problem is internal strife, civil war or revolution. It eventually destroys the institution and, if everything goes well, replaces it with a different one where at least the most blatant problems are fixed.

War or revoulution is not a desirable outcome though: In addition to the human suffering, it also tends to replace the people in power. But the people in power don't like to be replaced and so they will try to prevent it.

One manoevre they can use is to introduce planned institutional death: Every now and then the institution would be dismantled and created anew, without having to resort to a war or revolution.

Here's an example: The credit system tends to be one big suboptimal Nash equilibrium in itself. Compound interest grows the size of the debt like crazy and unless there's a way to limit the harm it'll destroy people and business and eventually the entire economy. Even lenders would be hurt, but none of them has a reason to mitigate the problem. They could, in theory, forgive the debt for the sake of keeping the economy afloat, but that would put them in disadvantage to other lenders.

And so the king or the religious authority decides to have jubilee years. Every fifty years, all debts are forgiven. The institution of money lending dies and is rises anew from the ashes. (David Graeber asserts that the practice was, in fact, not specific to Israel, but common at the time among the ancient societies in the Middle East.)

One can also think of the democratic system of regular elections as a kind of planned institutional death. Every four years, the government, with all the accumulated dysfunction, is thrown out and a new one is instituted. But the government example also makes the problem with planned death obvious. Government is replaced, but the people on non-political positions, various administrators and small-scale decision makers, remain. At least some inadequate Nash equilibria can therefore survive the change of the government. And those would accumulate over the time and eventually lead to the system collapse. We are between a rock and a hard place here: We want to destroy the institution to break the equilibria, but at the same time we want to preserve the institutional knowledge. We don't want to get all the way back to the trees after all. We don't want to get back to the middle ages either.

Last example that comes to mind is IETF, the institution that standardizes how Internet works. The real work, the development of standards, is done in working groups, which have a clear charter that defines what they are supposed to achieve and more importantly, how long would it take. The working group exists for, say, four months, and then dies. Sure, there are IETF institutions other than the working groups and those can survive for longer. But these are mostly doing the support jobs. Organizing meetings, publishing the new standards and so on. The real stuff happens in the working groups.

All in all, I am not at all sure that planned institutional death is a solution to all suboptimal equilibria problems, but the fact that evolution uses it, that it fights dysfunctions, such as cancer, by discarding the bulk of the cells every now and then and preserving only the germline, makes it at least worth of consideration.

June 26th, 2020

by martin_sustrik

Original article: Studies on Slack

1

Proposition: If you have enough free resources you can evolve irreducibly complex features.

Imagine a bunch of yeast cells added to a barrel of malt when beer is being brewed. They have an ocean of free resources at their disposal. Surely, the evolutionary pressure will be low and the yeast cells would be able to escape the local maximum and evolve an irreducibly complex eye.

Where this idea fails is not taking the nature of exponential growth into account. Exponential means being fast. It means, in fact, being faster than anything you can visualize. The barrel would be fully fermented in just couple of days.

In other words: No matter how big a whale fall is, it will be exploited fast. The body of the whale will be eaten in months and the last bacterial remains of the ecosystem will survive for maybe 100 years. Nowhere near the time needed for a complex evolutionary change to happen by chance.

It's worth looking at E. coli long-term evolution experiment here. Since 1988 E. coli is cultivated in a uniform substrate and observed to see how it evolves. From our perspective it's interesting that each day 1% of population is transferred to a new flask with the fresh substrate. One can think of it as of a new whale fall every day. Unfortunately, I am not aware of a control experiment where substrate would be added gradually, which would allow us to see whether environment with slack is more conductive to evolution of new features than environment without slack.

However, in one of the populations in the experiment a speciation event was observed. There's one strain of E. coli that has advantage during growth on the substrate (slack) and another strain that has advantage during stationary phase, when the substrate runs out (moloch).

That hints that supposed slack may not be slack at all, just a different environment exploited in competitive manner by different subset of species (ecological opportunists).

2

Proposition: Isolation of small subpopulations can produce slack.

Let's start with some basics of population genetics so that we have tools to understand what's going on.

Let's say no human being is able to roll their tongue. What happens when a mutation arises that gives one that ability?

Well, if rolling your tongue makes you more likely to survive and have more children then the trait will be actively selected for. The frequency of the tongue-rolling allele in the population will increase rapidly:

Note that when the frequency reaches 100% there's no way back. There are no more non-tongue-rollers that could reproduce and take over the population. We say that the allele becomes fixed.

But what happens if the allele has no effect on your fitness? What it tongue-rolling makes you neither more likely to survive and reproduce, nor less likely?

In that case the frequency of the allele is a random walk. If increases and decreases solely by chance. In most cases it means it will be wiped out quickly:

Sometimes, though, the allele gets lucky and becomes fixed:

We call this process of random fluctuation of allele frequency "genetic drift".

The things turn interesting when the population size is small. Imagine population of two. Now it's much easier for the new allele to become fixed. Once it comes to being via mutation it already has 50% frequency in the pool and all it takes to become fixed is reproducing once.

The smaller the population, the more alleles become fixed by chance alone. (The picture stolen from Wikipedia, CC BY-SA 3.0)

To put it in different words, there are two competing forces in play here. There's selection (determinism, Moloch). And then there's drift (randomness, slack). And there are two factors that determine which one is going to be stronger. First, the more effect the allele has on the fitness the more is its evolution driven by selection. Second, the smaller the population, the more it is driven by drift.

In very small populations even alleles that are deletrious (harmful to the survival of the individual) can become fixed by chance.

If there was a single allele that coded for the half of the irreducibly complex eye it could become fixed even though having a half of eye is, strictly speaking, worse than not having an eye at all.

What we gain here is time. We gain slack to wait for a new mutation to completes the eye. There are no more individuals with no eyes at all that would outcompete the half-eyed individuals. Now everybody has a half of an eye (even it the population size grows!) and we are waiting for something to happen. We are standing on a hillock in the adaptive landscape. It could be that the next mutation would produce no-eyed individuals again who would outcompete the half-eyed individuals. In that case we are back to where we started. But it may also happen that the next mutation would provide the second half of the eye and the individuals with the sense of sight would spread through the population.

Now, assuming that our small population originated by splitting from a large ancestral population, for example, some animals migrating to a new island, now that they have evolved a superior trait they can migrate back to their ancestral homeland, win the evolutionary race and become the dominant species. (See peripatric speciation.)

There's not much slack to be seen here. We can call small size of a population "slack" even if it is under intense evolutionary pressure, but that way the term loses its intuitive appeal.

But let's look futher.

Interesting alternative was proposed by the team of Andreas Wagner, here in Zurich. They start with the fact that neutral mutations (neither beneficial nor harmful) are subject to the random drift and thus a population is likely to have a lot of neutral variation at any given point of time. In other words, while the population has only a small variation of phenotypes (how the animals look like) it has large variation of genotypes (how the genomes of the animals look like). Then they show that different genotypes within the same phenotype can have vastly different "neighbourhoods". One animal may, to put it simply, undergo a single mutation and evolve an eye. Another may undergo a different mutation and evolve an ear.

Think of it as of a map. Each colour patch is a different phenotype. Locations within a patch are different genoptyes. Individuals in the population, thanks to neutral mutations (which are by definition free of the selection pressure) can explore the enire area of their particular phenotype. Once the environment changes there may be some individuals just across the border from the phenotype that would do better in the new conditions. Such individuals can undergo a single non-neutral mutation and result in a new phenotype which will then spread in the population via selection.

This may be closer to the intuitive notion of slack. Neutral mutations have slack because they don't affect the phenotype and are thus not acted upon by the selection.

3

Proposition: Two-layer evolutionary systems can produce their own slack. The outer system can introduce rules that give inner system some slack.

Let's look at some examples.

Multicullular organisms: Each cell is genetically identical to each other cell. There's no variance for the evolution to work with. Any mutations (cancer) are aggressively weeded out. There's no evolutionary slack in the sense of "freedom to evolve in random direction".

Eukaryotic cell: Eukaryotic cell has two genomes so it's harder to tell whether the outer system (nuclear DNA) gives the inner system (mitochondrial DNA) any slack. But we can make an educated guess. First, let's stress the point that mutations in mitochondrial DNA can be as dangerous as mutations in nuclear DNA. There is such a thing as mitochondrial cancer. It ends up with a cell packed up wall to wall with mitochondria. That implies that it doesn't pay off for the cell to give mitochondria too much slack. Second and more important: Look at the crazy measures the cells use to prevent mixing mitochondria from both parents when undergoing sexual reproduction. The fact that sperm cell is stripped of all of its mitochondria before the DNA is allowed to enter the egg cell hints that intra-cell mitochondiral evolutionary conflict is detrimental to the cell as a whole. To sum it up, it looks very much like eukaryotic cells give mitochondria as little slack as possible.

Social insects: In fully social insects such as bees, the individuals have different genomes, but they use a weird trick to ensure social cohesion. To make it short, their reproductive mechanism is such that a sister of a bee is more genetically similar to heself than her own offspring. That's an incentive not to reproduce herself but rather help the queen with reproducing. In the end the system doesn't look too much different from the system used by multicelular organisms. (It may be interesting to look at termites though, which lack the incentive system described above.)

All of the above are systems seem to be the cases of 100% coordination with no slack allowed.

4

Proposition: Two-layer evolutionary systems with migration can produce slack.

The difference to the previous scenario is that the lower level individuals (say, mitochondria) can migrate between higher level individuals (eukaryotic cells). While mitochondria cannot in fact migrate, there's a huge class of organisms that can. Namely, parasites. (In fact, it's a rule of thumb: Anything that cannot migrate tends to be a mutualist. Anything that can migrate has a tendency towards parasitism.)

These include typical parasites, such as pinworm, but also, on a different level, memes. The common characteristic is that the environment of a parasite is not a single individual of the host species, but rather the entire host species. In theory, an opportunity to play non-zero-sum games emerges: If a parasite helps its host to survive and reproduce it grows the host pool size and may get more populous itself. However, assuming free migration between hosts, there's nothing preventing the old bad strain of parasites invading the new hosts and eventually pushing the do-gooder parasites to the extinction.

5

Proposition: Two-layer evolutionary systems with controlled migration can produce slack.

I am going to get handwavy here.

Imagine that the control exercised by the outer system is incomplete. Not because the outer system would get any short-term evolutionary advantage from relaxing the control but simply because the controled individuals sometimes manage to escape and find a different host. It is not clear how such a relationship could be stable though. It seems that one side must prevail: Either the host gets full control and what we get is a mutually beneficial relationship or the symbiont manages to escape the control and becomes a hardmful parasite. (But look here.)

All that being said, there is one mechanism that seems to fit the bill. It's sex. Individual genes are able to escape the genome, but it happens in a very controlled manner.

Does that mean that sex gives species slack? One hint would be that sexual specifes in general evolve slower than asexual species. In other words, they are slacking. But I am not at all sure. If there's a mysterious question in evolutionary biology, it's the very existence of sex. Entire books were writen about it and there are dozens of proposed solutions. I don't feel confident enough to dig any deeper here, but it still may be worth considering.

May 23th, 2020

by martin_sustrik

Now, with billion people locked down in their homes, social contact over Internet becomes an increasingly important topic. Not only it allows people to stay in touch, it also lowers the incentives to leave one's home and meet people in person and thus contributes to the public health.

I threw an online birthday party few days ago and in this article I would like to share some of my observations about how a party over Internet differs from one in the physical world and point out some implications and possible improvements for videoconferencing software.

To start with, existing videoconferencing software is geared towards business meetings. And the differences between a business meeting and a party are easy to spot: At a party, there's much less structure. While at a meeting it's typically just one person that speaks and everybody else listens, at a party people tend to speak in parallel. At a party, non-verbal communication (facial expressions, gestures) is crucial. At a business meeting, not so much. At a meeting one wants to maintain self-control. At a party, one often rather wants to get rid of it. And so on and so forth.

So, let's move directly to my observations:

Everyone should be visible

Unlike with Google Hangouts where the person speaking is displayed in full-screen view and everyone else as a small icon on the side, at a party a want to see everyone in parallel. Sometimes, it's someone's non-verbal accompaniment of someone else's speech that's the real fun. But even when that's not the case, I still want to follow how everyone reacts to what is being said, whether they are laughing, paying attention, not paying attention, preparing a snack, drinking, smoking or whether they have left for a moment, leaving just a lonely chair visible in the window.

In short, I want to see all the participants of the party, side by side, all of them in equally sized windows. Size of the window is too expensive a piece of psychological real estate to waste on such an obvious thing as "who's speaking at the moment". More about that below.

Looking at people

Maybe the most unpleasant part of social videoconferencing is not being able to follow who's looking at whom. Everyone just stares, indiscriminately, at the camera.

I have no idea of how reliably the existing eye-tracking software works. After browsing the web for a minute it seems that eye tracking is used mostly for… optimizing ads??? What a waste of resources!

Anyway, it doesn't matter. Whether the software uses webcam to trace your gaze or whether you use mouse to explicitly point to whomever you are looking at the implications are more or less the same: First, I want the window I am looking at to get larger. Second, I want the person on the other side to know that I am looking at them - not necessarily in any obtrusive manner, I just want them to be aware of it, presumably by making my image slightly larger.

To give a practical example: One of the participants at the party wears an amusing mask. I point to the corresponding window which makes it larger and allows me to inspect the details of the mask. The person in question sees that I am looking at them and that, possibly, others are looking as well. They may choose to react to that.

Taken together, one would see large image of the person they are looking at, somewhat smaller images of people looking at them and small images of everyone else. That, I think, more of less reflects how people perceive each other in real-world social interactions.

Eye contact

If I look and someone and that person looks back at me, that's a powerful social signal and it should be reflected by the software. For example, in such a case we could get a special communication channel, where not only we see each other in large windows but also the talk by other people can be muffled so that we can hear each other well. (By the way, this protocol of looking and looking back is based on "cabeceo" as practiced when dancing tango: You can invite a person to dance with you by looking at them and they may accept by looking back or refuse by looking away.)

In this post I am not going to speak about larger parties but the problem there is obvious: With many people present there are going to be many parallel conversations, resulting it too much noise and not being able to hear each other properly. Clearly, some kind of fluent separation of the party into subgroups is needed. And while this seems to be a hard problem, the eye contact protocol described above can be used at least as a starting point: Cabeceo protocol allows to create groups of two people. Can it be somehow generalized for groups of three or more?

Kissing

In many cultures, kissing is an important part of social interaction. However, it doesn't lend itself well to communication over Internet.

Skype allows you to send an emoticon (e.g. a symbol of a heart) to your counterpart. But that doesn't really convey that feel of intimacy that the real kiss does. After all, emoticons were invented to be a substitute for non-verbal communication in the situations where people don't see each other (SMS, email) which makes them, in the context of videoconferencing, redundant at best and embarrassing at worst.

Obviously, sending a kiss through video can't be solved by purely technical means. It would require some social innovation. If Eskimos can kiss by rubbing their noses, why can't we devise a special Internet kiss? But if we do, the experience can be greatly improved by the software.

Consider the cabeceo protocol above. What if a kiss worked like this: I look at the other person. The other person looks at me. Our windows get larger, everyone else fades into background. Surrounding noises are muffled. Then we both touch cameras with our noses.

It would have to be tested in practice, but it kind of looks like it could feel quite intimate.

More social signals

In the physical world there are many more ways to send social signals.

Consider, for example, the seating order. If I sit next to someone it may mean that I want to speak with them, or I may want to signal that I belong to a certain subgroup or maybe it was just the only empty chair left. (Note that many social signals are weak and ambiguous. But that's a feature, not a bug!)

Now, I've used seating order just as an example. Not everything that exists at a physical party has to be necessarily mimicked in an online party. However, if the developers of the videoconferencing software decide that seating order is worth mimicking, then it has implications for the product design. For example, ordering of windows on the screen would have to be the same for everyone. It can't be that you can rearrange them arbitrarily. Should sitting next to someone else come with some extras? Maybe I can whisper to a person sitting besides me. Etc.

Miscellaneous

• There's a natural worry that seeing only people's faces is going to prevent a lot of non-verbal communication (gestures). However, my anecdotal evidence is that people, especially as the party progresses, subconsciously tend to move away from the camera making their entire upper body visible and thus improving non-verbal communication channels.

• Partying online is not all downsides. One upside is that, being at home, people have all kind of inventory at their disposal, which, at a meatspace party, they don't. In our case we've used a plague mask, a peacock feather, a bandana. There may be no technological implications, but I think the behavior is worth mentioning anyway.

• It's unclear whether you should see yourself among the participants of the party. On one hand, seeing yourself leads to more self-control, which is not that desirable at a party. On the other hand, people do like the feature and would probably feel awkward if it was removed. Feedback from a friend: "If I am not on the screen it would feel like I am not present at the party."

Conclusion

These are just some random thoughts that I'm giving free to anyone who fancies to implement them. However, if you do, please do let me know! It would be really interesting to see how this works in practice.

April 5th, 2020

by martin_sustrik

In computer science, compiler is a program that translates things that programmers write, such as "if CloseButtonPressed then CloseApplication", to things that computers are able to understand and execute, such as "0001010111001010111011000111100010101011110".

When the first compiler was written, someone had to write it in the machine code, the zeroes and ones that computers understand. They had to type in the numbers, or, more likely, punch holes into punchcards.

But from that point on, programmers were able to write human-readable programs instead of the inscrutable machine code. They just had to use the existing compiler to translate from the former to the latter.

Now imagine you are tasked with writing the second, improved, version of the compiler.

You can, of course, do the same thing again. You can type in the numbers, or maybe punch zeroes and ones into punchcards. But writing machine code is hard.

And now you have a better option at your disposal: You can write the new version of the compiler in a human-readable language and use the old version of the compiler to translate it into machine code!

Similarly, the third version of the compiler can be compiled by the second version of the compiler, the fourth by the third and so on.

That, by the way is not a hypothetical scenario. That's how the compilers are written nowadays.

Now, here comes the question: What if all the software disappeared overnight? What if the only thing we were left with was the human-readable source code of the latest version of the compiler? Would we be able to reconstruct the language and the compiler and get everything running again?

On one hand, the source code contains all the relevant information. It contains the full description of the language, its syntax and its semantics. It contains all the instructions needed to translate it into the machine code. It should be therefore possible to use it to create a functional compiler again.

But on the other hand, how exactly would you do that? The source code is, after all, just a pile of text. And there's no compiler to turn it into the machine code, into something that computer understands. It sounds like you are faced with a kind of chicken-and-egg problem.

Clearly, the source code is not enough. There's some information missing. But it's hard to pinpoint what exactly the missing part may be.

***

What if we found an undamaged T-Rex DNA in the gut of ancient mosquito trapped inside a piece of amber? Would we be able to grow an adult T-Rex? Would we be able to find out what colour they were, whether they had feathers and whether males attracted females by nightingale-like song?

On one hand, one regularly reads news about scientists being on the verge of resurrecting the wooly mammoth. It am no embryologist, but as far as I understand, the idea is that the DNA reconstructed from the frozen remains of mammoths found in Siberia would be injected into an elephant egg and grown either in an elephant female or maybe ex-vivo.

On the other hand, dinosaurs have no close living relatives. For T-Rex, there's no equivalent of what elephant is to mammoth. Maybe the DNA could be injected into an ostrich egg, the birds being the closest surviving relatives to dinosaurs. But given the evolutionary distance between T-Rex and ostrich, how likely it is that the biochemistry and embryology of the ostrich egg is similar enough to the biochemistry and embryology of the T-Rex egg to produce a viable offspring? I wouldn't bet much on it.

And even if that was possible, what if there was no living organism left, just a piece of DNA? Would we still be able to reconstruct the animal?

Although DNA encodes all the information about the organism, there's clearly something missing. And, again, it's hard to put your finger on what exactly it may be.

***

Central African economy is failing badly. CAR authorities therefore look around for inspiration and decide to adopt the Swiss model. Switzerland, after all, is a small multi-ethnic, multi-confessional country, just like CAR, and it was able to raise from being an obscure and poor backwater country to one of the world's most wealthy countries in just few decades.

CAR therefore adopts the Swiss constitution, Swiss political system, Swiss law, Swiss approach to bank secrecy, Swiss educational system. They establish a church choir in every village and start eating fondue.

But to everyone's surprise nothing changes. CAR economy is doing as badly as ever. Corruption is rife. The rule of law falters. The institutions that run like clockwork in Switzerland fail mysteriously in CAR.

Clearly, there's something about Swiss society that wasn't captured in the laws and institutions introduced in CAR. But, once again, it's hard to say what the missing piece could possibly be.

October 27th, 2019

by martin_sustrik

I don't care much about feast days or memorial days. Assumption of Mary? Meh. Slovak Constitution Day? Shrug. Labour Day? Real socialism had sucked all the joy out of that one.

The only one that I do observe is Petrov Day. Not that I do anything special. I don't even take a day off from work. I just take few minutes to contemplate.

But now I am thinking that maybe, on this festive occasion, I should do a little bit more, maybe something that's public, something that has at least a slight feeling of a ceremony.

And given that while browsing the newspapers today I haven't found a single mention of the event, no op-eds, no commemorative articles, no historical analysis, I decided to write a witness report of my own, no matter how petty and unimportant it may be.

On September 26th, 1983 I was ten years old. I was attending 4th grade of the Czech school by Czechoslovak embassy in London. The authorities didn't want socialist kids to mingle with capitalist kids, so we weren't allowed to attend local schools. (There were sporadic, unsuccessfull, attempts to mix the kids from different socialist countries though.) The school resided in a family house in Hampstead and there were approximately forty kids attending. I don't remember that particular day, but given that it was in autumn and that it was in London, I assume it was raining. It was Monday. We used to live close to the school, several Czechoslovak families close to each other and so, same as every morning, one of the mothers walked us to the school. Kids that lived further away were driven in by a minibus. There were four grades in the school, two teachers and two classrooms. Kindergarten was in a separate room near the entrance. Me and my friend, we once did a puppet show for the kindergarteners. We had only one puppet (a chicken) and so I have no idea what the other one of us was doing. The memories have faded in the meantime. There was a narrow room with no windows in the basement that served as a library. My favourite book was about cars: A hand-illustrated compendium of models from the very beginning of automobilism up to the seventies. One of the cars was called Zaporozhets which was a bit puzzling and mysterious because it sounded like it should have horns. The room was also used for more nefarious purposes, when the boys were all too interested in what's under girls' skirts and the girls were not unwilling to give them a peek. In the afternoon, back at home, we used to play with Lego and my new chemistry set. Mostly mixing all chemicals together and obtaining a boring greyish liquid. We even had an experimental vegetable garden where we successfully grew carrots 2 cm in length. Me and my sister, we used to sleep in a bunk bed. When she got angry she used to hit me with a book called "Fish and Fishing". I was older and had longer hands though, so I was able to keep her at a safe distance.

And yes, we did our homework because we thought the next day was coming. We had no idea that we were almost burned alive that day.

September 26th, 2019

by martin_sustrik

It is said that every year the IQ needed to destroy the world drops by one point.

Well, yes, but let me add a different spin on the problem:

Every year, the IQ needed to make sense of the world raises by one point.

If your IQ is 100 and you want to see yourself in 2039 just ask somebody with IQ 80 and listen carefully.

I know that some people are troubled about prospects of those less intellectually gifted in the modern knowledge-based economy. And yes, it's troubling that we are heading towards some kind of intellectual elitism. But, on the other hand, it may be just a temporary thing. At the end we will all, village idiots and von Neumanns alike, end up having no clue.

What are we going to do then?

September 24th, 2019

by martin_sustrik

Since writing the post on a hypothetical hull language as an alternative to shell I cannot stop thinking about the shortcomings of shell.

And one think that comes to mind over and over is type-safeness. Shell treats everything as a string and that's the source of both its power and its poor maintainability.

So when I ask whether shell can be improved, the question is actually more subtle: Can it be improved without compromising its versatility? Can we, for example, be more type-safe without having to type Java-like stuff on the command line? Without sacrificing the powerful and dangerous features like string expansion?

I mean, you can write shell-like scripts in Python even today and use type hints to get type safeness. But in real world this practice seems to be restricted to writing more complex programs, programs that require actual in-language processing, complex control flow, use of libraries and so on. Your typical shell script which just chains together a handful of UNIX utilities — no, I don't see that happening a lot.

To put it in other words, different "scripting languages" managed to carve their own problem spaces from what once used to be the domain of shell, but almost none of them attacked its very core use case, the place where it acts as a dumb glue between stand-alone applications.

But when writing shell scripts, I observe that I do have a type system in mind. When I type "ls" I know that an argument of type "path" should follow. Sometimes I am even explicit about it. When I save JSON into a file, I name it "foo.json". But none of that is formalized in the language.

And in some way, albeit in a very hacky one, shell is to some extent aware of the types. When I type "ls" and press Tab twice a list of files appears on the screen. When I type "git checkout" pressing Tab twice results in a list of git branches. So, in a way, shell "knows" what kind of argument is expected.

And the question that's bugging me is whether the same can be done in a more systemic way.

Maybe it's possible to have a shell-like language with actual type system. Maybe it could know that file with .json extension is supposed to contain JSON. Or it could know that "jq" expects JSON as an input. Maybe it could know that JSON is a kind of text file and that any program accepting a text file (e.g. grep) can therefore accept JSON as well. And it could know that "ls -l" returns a specific "type", a refinement of "text file" and "file with one item per line", with items like access rights, ownership, file size and so on.

But how would one do that?

In addition to the language implementing a type system it would require some kind of annotation of common UNIX utilities, adding formal specification of their arguments and outputs. (With all programs not present in the database defaulting to "any number of arguments of any type and any output".) Maybe it can be done by simple type-safe wrappers on top of existing non-type-safe binaries.

There seems to be a lot of food for thought here.

May 12th, 2019

by martin_sustrik

Here's an idea. It's kind of crazy but it's also fun to think about.

Imagine a language which has every piece of its state stored in the filesystem.

Yes, I know it sounds weird but do read on!

Datatypes

Let's say we do this:

a = 1

What will happen under the covers is that file "a" will be created that contains nothing but "1".

Lists could be represented as files where each line corresponds to an item.

b = [1, 2, 3]

The above would simply create file "b" with the following content:

1 2 3

Filesystem also provides hash tables out of the box. They are called directories.

c = { foo: 1, bar: { quux: 3 } }

Which would create a directory "c" with file "foo" (containing "1") and directory "bar" containing file "quux" which would in turn contain "2".

Fun idea: For map indexing use operator / instead of []. So, to access "quux" you would write "c/bar/quux" which happens to be the path to the file in question.

Call stack

Of course, state of the program is not just about variables. We would have to represent the call stack in some way.

Easy to do! For each stack frame create a directory called "nextframe". The call stack would then look like a sequence of embedded directories:

main |-- a # contains 1 |-- b # contains 1\n2\n3 `-- nextframe |-- c | |-- foo # contains 1 | `-- bar | `-- quux # contains 2 `-- nextframe `-- a # contains 2

In such a model, scoping is easy. If you want to access "b" in the innermost frame, just walk up the directory tree until you find a file (or directory) named "b".

Also, overrides work as expected. If you ask for "a" in the innermost frame, you'll get "2", not "1".

Instruction pointer

Now you are probably wondering how would one persist the instruction pointer. The language needs to know which command is the next to execute after all and not making this information persistent defeats the entire point of the language.

Let's make it work this way:

When program is started, a copy of the source file is created. We'll call it a "todo" file.

Then the first line of the "todo" file is executed. Then it is removed from the file.

We'll do the same with the second and every subsequent line.

That way the "todo" file always contains a list of commands that are yet to be executed.

But hey, you cry, if there's a loop in the source code we can't just delete a command within the loop! We will still need it in the next iteration!

Easy to fix! Let's make each iteration of the loop create a new stack frame (we would want that anyway, to get proper variable scoping). The body of the loop would be then copied to the "todo" file of the new frame and executed. Once there's nothing left in the inner "todo" file the scope would be exited, the stack frame deleted and the parent could start new iteration of the loop with a new copy of the loop body. Finally, when there's no more iterations to do the parent will delete the entire loop construct, including the loop body and move on.

Let's have a look at an example:

b = [1, 2, 3] for i in b: echo i

Once the first line is executed we end up with the following state on the disk:

main |-- b # contains "1\n2\n3" `-- todo # contains "for i in b:\n echo i"

First iteration of the loop is started. Interpreter creates a new stack frame and populated both the "i" variable and the local "todo" file:

main |-- b # contains "1\n2\n3" |-- todo # contains "for i in b:\n echo i" `-- nextframe |-- i # contains "1" `-- todo #contains "echo i"

Now the echo command can be executed which will print "1" and remove the statement from the "todo" file:

main |-- b # contains "1\n2\n3" |-- todo # contains "for i in b:\n echo i" `-- nextframe |-- i # contains "1" `-- todo #contains ""

"todo" file is empty now so we can exit and delete the scope:

main |-- b # contains "2\n3" `-- todo # contains "for i in b:\n echo i"

Note how the "for" construct deleted the first element of "b". This may or may not be a good idea, but it's simple, so let's go with it for now.

At this point we can do the entire dance described above for element "2", then for element "3".

Finally, there are no more elements in "b" and "for" construct is considered done. It can be deleted from the "todo" file.

main |-- b # contains "" `-- todo # contains ""

As there's nothing more to do, the entire program can now exit.

Why the hell?

I mean, it's a fun exercise, but why would anyone want to use such a language?

But once you start playing with concept a little you'll find out that it has some interesting properties.

Debugging

For starters, it kind of feels like being in a debugger although there's no debugger anywhere in sight.

It's entirely possible to execute just one statement at a time. The entire state of the program is on the disk, so once you've executed one command you can execute the next one and so on.

In the process you can inspect all the variables: Remember? They are just files in the filesystem.

$ hull next done: b = [1, 2, 3] $ cat b 1 2 3

Time travel

Given that the entire state of the program resides in a directory, you can backup the directory and proceed with the debugging. Then, when you want to return to the previous point, you just restore it from the backup and you are set to go.

Laziness

Once again: Given that the entire state of the program is stored on the disk, it means that you can execute it partially, then do something else, then resume the execution.

For example, a parent program may be interested only in the first line of your program's output. It can run the interpreter until the first line is produced, then suspend it and go on with it's own stuff. When it needs the second line it would just resume your program and so on.

In this way it works very much like python generators. Or, for that matter, like unix pipes.

And it's also a bit like Haskell: No code has to be executed unless you actually need the results.

Concurrency

Given that programs are interruptible, you can launch many of them and then schedule them as you wish. The scheduling can be driven by commands such as "give me the next line of output".

To put it in a different way: Forking is easy. Just copy the directory and run second instance of the interpreter.

Here we are getting into the area of CSP, goroutines and channels.

Remote execution

Want to move the program to a different machine?

Easy! Just scp the directory to the machine in question. Then ask the interpreter to resume the execution where it has ended.

April 28th, 2019

EDIT May 4th 2019: Typesystem based on file extensions, including polymorphism: frobnicate(foo.json) can have different implementation than frobnicate(foo.xml)

by martin_sustrik

What are we doing when designing a programming language? We decide whether it's going to be imperative or declarative. We add a bunch of operators. We add some kind of objects and visibility rules. We decide to make it either strongly-typed or weakly-typed. Maybe we add generics or inheritance, or maybe multiple inheritance. And so on.

The question that interests me is whether the nature of these tools is determined by the problems we want to solve, whether they are, in some way, inherent to or, in other words, directly inferrable from the nature of the problem at hand, or whether they are rather crutches for our imagination, a purely psychological constructs that help our imperfect brains to deal with the complexity of the real world.

If you asked me ten years ago, I would probably argue for the latter.

And it's not hard to "prove" it: If two people write code to solve the same problem and one makes a terrible spaghetti monster in COBOL while the other goes for super-elegant and highly abstracted solution in Haskell, does it really matter to the computer? As long as the two are compiled to the same machine code, the machine does not care. All the clever constructs used, all the elegance, they are there only to guide our intuition about the code.

So, you see, the design of a programming language is determined by human psychology, by the quirks and foibles of the human brain. If there are inheritance hierarchies in your language it's not because things in the real world tend to arrange in neat trees — in fact, they never do, and if you want to point out the tree of life, I have bad news for you: most organisms have two parents and the tree of life is really a DAG of life — it's because human brains like to think in terms of taxonomies. They just happen to find it easier to deal with the complex world in that particular way.

But is it really so?

If you asked me today I wouldn't be so sure.

But it's a question that is not easy to answer. If you wanted to really tackle it you would need to take human brain out of the equation, so that you can find out whether the same kinds of "language constructs" arise even without a brain being involved.

So here's the idea: What about genetic algorithms? They don't require a programmer. Yet we can look at them and determine if a language feature, say encapsulation, does emerge from the process.

Or, rather, given the relative scarcity of data on genetic algorithms, let's have a look at evolution by natural selection. There's a code (genetic code), just like any programming language it can be executed (thus producing a phenotype) and there's definitely no brain involved. And we can ask: Is it just a random mess of instructions that happens, by a blind chance, to produce a viable phenotype or is there any internal structure to the code, something that we would recognize as a feature of a programming language?

And, I think, the answer may be yes.

Let's have a look at the concept of "evolvability". What it says is that natural selection may, in some cases, prefer individuals who have no direct, physical advantage, but whose progeny is more likely to adapt well to the changing evolutionary pressures.

But what does that even mean? Well, here's a concrete example:

Imagine two gazelles that are, phenotypically, the same. They look exactly the same, they behave the same etc. One would naively expect that neither of them would be preferred by natural selection.

But consider this: One of them has the length of the left rear leg encoded in one gene and the length of the right rear leg in a different gene. The other has the length of both hind legs encoded in a single gene. If that gene mutates both legs will be either longer or shorter, but they will never have different lengths.

Which of them is going to fare better in the race for survival?

I would bet on the single-gene one. When the environment changes and demands longer hind legs the evolutionary process would remain blind, some of the offspring would have longer legs and some of them would have shorter legs, but at least there wouldn't be the need to invest precious resources in all those unhopeful mutants with left leg shorter than the right one.

What the concept of evolvability says is that representation matters. The encoding isn't, in the long run, selectively neutral.

And as programmers we have a direct equivalent of the above: We call it subroutines. If the code is doing the same thing at two places, it's better to create a single subroutine and call it twice than to make two copies of the code.

Yet another example: Do you think that an organism in which every gene effects every part of its phenotype, the nose, the tail, the metabolism, the behaviour, is going to fare better than an organism where each gene is specialized for a single concrete task?

No, it's not. Every feature depending on every gene means that every mutation is going to change the phenotype significantly, in multiple ways. It's very likely that at least one of those changes is going to be deadly.

And again, there's a clear counterpart to that in programming languages. It's called modularity or, if you wish, encapsulation.

So, in the end, it seems the representation, the language features matter even if there's no human brain around to take advantage of them.

That is not to say that some of the language features aren't purely brain-oriented. Sane variable naming, for example, is likely to be such feature.

But still, at least some of what we encounter in programming languages probably goes deeper than that and is, in fact, objectively useful.

Now, to end with a lighter topic: Although I am a big fan of Stanisław Lem and therefore I have pretty serious doubts about whether we'll be able to communicate with the aliens if we ever meet them (having different society, different biology, different brain, different everything is not going to make it easy) the reasoning above gives us at least some hope. If both we and the aliens write computer programs they are probably going to share at least some features (subroutines, modularity). Calling that kind of commonality "understanding" may be an exaggeration but it's still better than nothing.

April 22nd, 2019

by martin_sustrik

Few days ago I've read an article in the local newspaper about Switzerland considering to lower the voting age to 16.

The reason I found it interesting was that it was not one of the old tired political discussions supported by the same old tired arguments that you typically encounter. In fact, it's a question that I have never thought of before.

Apparently, the discussion was triggered by the recent school strike for climate that went quite big in Switzerland. I've attended the demonstration in Zurich and it was not only big, it was really a kids' event. You could spot a grown-up here and there but they were pretty rare. (Btw, I think this movement is worth watching. Here, for the first time, I see a coordination on truly global level. It spans beyong western countries, with events being hosted in Asia, Pacific Islands, South America or Africa.)

Anyway, the main argument for lowering the voting age is to counter-balance the greying of the electorate.

Once again, this stems from what the climate stikers say: "The politicians who decide on these issues will be dead by the time the shit hits the fan. It will be us who'll have to deal with it. We should have a say in the matter."

But the question is broader: As the demographics change, with the birth rates dropping at crazy speed (China's population will start shrinking not that far in the future; Sub-saharan fartility rates had plummeted from 6.8 in 1970's to 4.85 in 2015), the age pyramid is going to look less like a pyramid and more like a column or even a funnel. In such a case the old will hold a much larger amount of political power than they do today.

While that may seem like a minor thing (everyone is young at some point and old later on) just consider how it would affect the politics of, say, pensions or health-care.

Or, for that matter, I hear that Brexit wouldn't happen is 16- and 17-year olds were allowed to vote.

More questions:

With old people being generally more conservative are we going to see slowing or even reversal of the seemingly instoppable move to the political left that was going on for decades?

With high percentage of young males being often blamed for social unrest and wars, is the changing shape of the age pyramid going to result in even more political stability? And how is giving teenagers a vote going to affect that?

I have no answers but the topic is definitely worth thinking about.

(Btw, the voting age was lowered to 16 in canton Glarus in 2007, so there's more than a decade of data to analyse the impact of the measure.)

March 24th, 2019

by martin_sustrik

When Black Panther came out I went to cinema hoping to finally see a science fiction movie about Sub-Saharan Africa. (District 9, for obvious reasons, doesn't count.)

Needless to say, I was disappointed. The film wasn't really about Africa. It was about America, pretending, half-heartedly, to be about Africa.

I was not sure how to feel about it. On one hand — trying to relate it to my own cultural background — a Hollywood movie about Slovak shepherds with laser weapons would be hilarious. On the other hand, it would be a wasted opportunity. All the stangeness and complexity of Slovak rural life would be thrown out, leaving only superficial, schematized image of an Eastern European country, presumably a mix of Dracula and Hunt for Red October.

Sub-Saharan Africa is much more complex and strange than boring Slovakia, but all we've got were some cringe-worthy depictions of local customs and a skyscraper that kind of looked like the mosque of Djenne.

For exactly the reasons above I loved Liu Cixin's science fiction novels. They are distinctly non-American and, for that matter, non-European. They are unapologetically Chinese and thanks to that they are often shocking for the western reader. What left a lasting impression on me, personally, was their bleak worldview, the total resignation on the possibility of cooperation, the "it's either us or them" attitude, where neither party is particularly evil, yet nobody seems to be able to imagine a world where the civilizations would treat each other with anything better than genocide.

Muqaata'a by Fahad Himsi (self-published, 2017) is another such book. Not only does it have little in common with either American or European science fiction, it doesn't really intersect with them. It's doing its own thing and it doesn't seem to care about what goes under the "science-fiction" label elsewhere in the world.

It is alien.

One wouldn't even call it science-fiction if the visitors from space weren't part of the story.

However, unlike with The Three-Body Problem, I can't imagine it ever becoming popular. Not only is the name of the novel impossible to remember. It's not even that it contains no less than three sounds that don't exist in English. It's also the lengthy discussions of Middle Eastern politics that are going to bore every science-fiction fan to death. And while there are undoubtely people who would find that kind of content interesting, those are unlikely to reach for a book from science-fiction shelf.

Despite all that, the book is interesting and worth reading.

It's a first contact novel and its premise is that the aliens, instead of touching down safely at the lawn in front of Palace of Nations, Geneva, managed to land somewhere in the war-torn provinces at Syrian-Iraqi border.

The book begins with a sequence of press clippings.

It's immediately clear that nobody has any idea of what's going on. Press agencies have no reporters on the ground. Everybody is guessing and spreading second- or third-hand rumors.

But what really stands out is the repressed anxiety that permeates the clippings.

Sometimes it's just reminding the reader of what's going on in Syria, the beheadings, the slavery, the kids being gassed, and asking a simple question: "What are They going to think about us?"

At other times authors get closer to the root of the issue: The aliens are unlikely to spot a difference between a bearded Islamic State fighter and a Nobel Peace Prize winner. Whatever is happening in Syria is going to be attributed to the humanity as a whole.

As one op-ed puts it: "You invited guests for the dinner. And while you are waiting, wearing your best tuxedo, lighting candles and switching on chamber music, the guests somehow miss the front porch and instead knock on the door of the cellar where you keep your wife and children locked away."

It feels like puncturing an ulcer. All the topics that are taboo in polite society, all the long avoided thoughts about starving children in Yemen and Horn of Africa are suddenly dragged into the daylight by the frightening fact that the humanity, for the first time in its history, no longer lives in the good old lord-of-the-flies world, but is suddenly placed under direct adult supervision.

Himsi is playing with the concept of collective guilt.

If Polish government outlaws calling death camps located in Poland "Polish death camps" arguing they were really German death camps, then you, as a Pole, may feel better for a while. That is, until you realize that in addition to being German death camps they were also human death camps and if you accept the idea of collective guilt then you, as a human being, are directly responsible for everything that happened there. Collective guilt cuts both ways. You can, obviously, escape the realization by relaying on tribal parts of your brain, but Himsi cleverly smashes that particular psychological machinery by putting aliens, unable to distinguish between a German and a Pole, into the story.

(Speaking of collective guilt, David Brin's Uplift series deserve a mention. It would be a fairly standard space opera if not for its treatment of the species as moral agents: If galactic community knew that humans exterminated the giant sloth we would be held responsible. And people are, needless to say, working hard on covering all the traces.)

But back to Himsi's book: If you'd expected humanity to go through some kind of moral catharsis, you'd be wrong.

It doesn't take longer than couple of weeks until the self-questioning voices are drowned in the noise of conspiracy theories, us-versus-them narratives and stories that were undoubtely written under the influence of LSD.

Sabah writes about aliens supplying PKK with annihilation weapons. The very next day Hürriyet publishes a toned-down version of the same story. Politico features a long-winded essay on impact of alien presence on the global balance of power. Right-wing tabloids worry: "Are the aliens going to convert to militant Islam?"

And amid of all the flowing puss the first part of the book begins. Himsi offers you a deal. If you can fight the nausea there's a job to be done. And maybe things to be learned.

And the quest Himsi invites you to take part in is to explore the political implications of aliens in the Middle East.

March 10th, 2019

by martin_sustrik

After working some more with Tiles, my mini-library for programmatic code generation, I've realized it has a feature that I haven't knowingly baked in, but which makes it a really good tool for the job. Namely, the code you write is composable.

Here's what I mean: If you want to generate a C function that prints out names of countries' capitals then you can generate the code that does the actual printing first:

data = { "Afghanistan": "Kabul", "Brazil": "Brasilia", "Canada": "Ottawa", } body = t/'' for country, capital in data.items(): body |= t/""" printf("@{capital} is the capital of @{country}"); """

Once done, you cen generate the scaffolding:

program = t/""" #include <stdio.h> int main(void) { @{body} return 0; } """

The example, of course, shows just two levels of nesting. In practice, however, you'd have many levels of nesting.

But whatever the nesting level, you, as a programmer, can focus on a single task at a time. First, you think about how to print the capitals. Then you think about the scaffolding. There's no overlap.

Compare that to your typical templating solution:

template = Template(""" #include <stdio.h> int main(void) { {% for country, capital in data.items() %} printf("{{ capital }} is the capital of {{ country }}"); {% endfor %} return 0; } """) template.render(data=data)

Note how you start with writing scaffolding (include, main) then you switch to the business logic (printf), then you get back to scaffolding (return).

The example is trivial and as such, the both solutions are kind of comparable.

However, imagine there are five levels of nesting.

The code written using Tiles would, logically, look like this:

level1 level2 level3 level4 level5

The code written using one the classic templating tools, on the other hand, would look like this:

level1 level2 level3 level4 level5 level4 level3 level2 level1

March 2nd, 2019

by martin_sustrik

Now, I know that this is super-weird and probably makes sense to nobody by myself.

First, it's in lydian scale which is nowadays used nowhere but in some Slovak (and, allegedly, also in Polish) folk songs.

Second, the base chord is dissonant (C maj).

It's meant to be played by strings so that dissonances do not fade out.

Use as you see fit.

Feb 25th, 2019

by martin_sustrik

Writing programs that generate programs is hard. The programmer has to think at two levels of abstraction at once. She has to follow the logic of the generator. At the same time she can't lose the focus on the logic of the generated code. And the two don't even have to be written in the same language!

That's a hard enough feat even when the tools aren't putting obstacles in your way. But, unfortunately, that's exactly what they are doing.

Consider this Python program that outputs the classic C "Hello, world!" program:

def generate_hello(who): return """#include <stdio.h> main( ) { printf("hello, """ + who + "!\\n\");\n}" print(generate_hello("world"))

Ugly, you say? Yes, it's ugly. But it's just generating the simplest possible program! If we wanted to generate something truly complex it would become doubleplusugly.

But ugliness aside, the problem is that the code is unreadable.

Reading code is generally harder than writing code. Reading code with two parallel levels of abstraction is yet much harder. Add some atrocious formatting, sprinkle with copious amount of escape sequences and even the best programmer won't be able to understand what's going on.

The traditional solution to this problem is templating.

The idea is that the generated program is like a form, a pre-printed template with few blank slots to fill in:

And here's how it works with, say Jinja2:

from jinja2 import Template t = Template("""#include <stdio.h> main( ) { printf("Hello, {{ who }}!\\n"); }""") print(t.render(who="World"))

Well, it's not much better. Weird formatting and escape sequences remain. However, given that the template is now a single string we can load it from a file instead of using a string literal. The content of the file would look much better:

#include <stdio.h> main( ) { printf("Hello, {{ who }}!\n"); }

The downside is that the template and the generator now live in two different files which makes the logic harder to follow.

By the way, I am not picking on Jinja2 here. All the code generation tools I've looked at work in basically the same manner. Here's, for example, how Golang's text/template package looks like:

package main import ( "log" "os" "text/template" ) func main() { // Define a template. const letter = ` Dear {{.Name}}, {{if .Attended}} It was a pleasure to see you at the wedding. {{- else}} It is a shame you couldn't make it to the wedding. {{- end}} {{with .Gift -}} Thank you for the lovely {{.}}. {{end}} Best wishes, Josie ` // Prepare some data to insert into the template. type Recipient struct { Name, Gift string Attended bool } var recipients = []Recipient{ {"Aunt Mildred", "bone china tea set", true}, {"Uncle John", "moleskin pants", false}, {"Cousin Rodney", "", false}, } // Create a new template and parse the letter into it. t := template.Must(template.New("letter").Parse(letter)) // Execute the template for each recipient. for _, r := range recipients { err := t.Execute(os.Stdout, r) if err != nil { log.Println("executing template:", err) } } }

It code generation was really like filling in tax forms, the templating approach would be good enough.

But it's not.

Even the simple Go example above requires some extra logic. Specifically, it uses different text depending on whether the person is question attended the wedding or not.

But it gets worse.

Imagine you want to generate the following report:

ACME Inc. - Payroll Alice $3000 Bob $2500 Carol $2800 Dylan $2900 Signature: .......... (Wile E. Coyote)

That's no longer filling in empty slots. The template has to loop through the list of employees and generate a line for each of them.

Again Jinja2:

{{ company }} - Payroll {% for employee in employees %} {{ employee.name }} ${{ employee.salary }} {% endfor %} Signature: .......... ({{ director }})

As can be seen, moving to more complex generated text means adding more special constructs (if-then-else, for-in etc.) into the template until we end up with a full, Turing-complete code generation DSL.

There are two things I don't like about the templating approach to code generation:

First, I don't want to learn a new DSL. If I'm coding in Go, I want to use Go and I want to use its power fully. I don't want to use some crippled version of it that's used only inside of templates.

Second, I don't want to use large templates in the first place. The pieces I want to fill into the template are often too complex to be generated inside of the template and therefore I have to precompute them, put them in an array and then use text/template to render the array. That in turn tears the generation logic, which is a single conceptual thing, into two pieces: The precomputation and the template rendering.

And I am not even speaking of complex cases when I want to fill in a slot in a template not by a simple string but rather by a full template of its own.

Given the considerations above, I've created a small Python library in 2017 to do code generation in a different way. The idea was to treat "a block of code" as a primitive type, not unlike string or integer. The user would then use the language — the real language, not a limited version thereof — to manipulate those blocks of code, add them together and eventually generate the entire program out of them.

payroll = t/"@{company} - Payroll" for employee in employees payroll |= t/"@{employee.name} $@{employee.salary}" payroll |= t/"Signature: .......... (@{director})" print(payroll)

As can be seen, it's a pure Python. No template-specific DSL, no nothing. Instead, there's a new primitive type called "tile" that really just a rectangular area of text. Tile literals support tile interpolation (@{} stuff) but that's not much different from the existing Python string interpolation (F-strings) and can't be really claimed to be a separate language within a language. The user is free to manipulate the tiles in any way that she sees fit.

Unfortunately, since 2017, I haven't had a chance to use the library in anger.

Until last week, that is.

Doing so resulted in smoothing of the API and adding of some convenience features. It also bloated the code of the library from 35 LoC to 74 LoC (!)

The code I have written generates some man pages and C header files.

Let me paste a snippet here so that you get a feeling how a real-world code written with tiles looks like:

# SYNOPSIS section synopsis = t/'#include<@{fx["header"]}>' if fx["add_to_synopsis"]: synopsis |= t%'' | t/'@{fx["add_to_synopsis"]}' synopsis |= t%'' | t/'@{signature(fx)}' # DESCRIPTION section description = t/'@{fx["prologue"]}' if fx["protocol"]: description = t/'@{fx["protocol"]["info"]}' | t%'' | description if fx["experimental"] or (fx["protocol"] and fx["protocol"]["experimental"]): description = t/'**WARNING: This is experimental functionality and the API may change in the future.**' | t%'' | description if fx["has_iol"]: description |= t%'' | t/""" This function accepts a linked list of I/O buffers instead of a single buffer. Argument **first** points to the first item in the list, **last** points to the last buffer in the list. The list represents a single, fragmented message, not a list of multiple messages. Structure **iolist** has the following members: ```c void *iol_base; /* Pointer to the buffer. */ size_t iol_len; /* Size of the buffer. */ struct iolist *iol_next; /* Next buffer in the list. */ int iol_rsvd; /* Reserved. Must be set to zero. */ ``` When receiving, **iol_base** equal to NULL means that **iol_len** bytes should be skipped. The function returns **EINVAL** error in the case the list is malformed: * If **last->iol_next** is not **NULL**. * If **first** and **last** don't belong to the same list. * If there's a loop in the list. * If **iol_rsvd** of any item is non-zero. The list (but not the buffers themselves) can be temporarily modified while the function is in progress. However, once the function returns the list is guaranteed to be the same as before the call. """ if fx["args"]: description |= t%'' | (t/'').vjoin([t/'**@{arg["name"]}**: @{arg["info"]}' for arg in fx["args"]]) if fx["epilogue"]: description |= t%'' | t/'@{fx["epilogue"]}' if fx["protocol"] or fx["topic"] == "IP addresses": description |= t%'' | t/'This function is not available if libdill is compiled with **--disable-sockets** option.' if fx["protocol"] and fx["protocol"]["topic"] == "TLS protocol": description |= t%'' | t/'This function is not available if libdill is compiled without **--enable-tls** option.' # RETURN VALUE section if fx["result"]: if fx["result"]["success"] and fx["result"]["error"]: retval = t/""" In case of success the function returns @{fx["result"]["success"]}. 'In case of error it returns @{fx["result"]["error"]} and sets **errno** to one of the values below. """ if fx["result"]["info"]: retval = t/'@{fx["result"]["info"]}' else: retval = t/"None." # ERRORS section standard_errors = { "EBADF": "Invalid handle.", "EBUSY": "The handle is currently being used by a different coroutine.", "ETIMEDOUT": "Deadline was reached.", "ENOMEM": "Not enough memory.", "EMFILE": "The maximum number of file descriptors in the process are already open.", "ENFILE": "The maximum number of file descriptors in the system are already open.", "EINVAL": "Invalid argument.", "EMSGSIZE": "The data won't fit into the supplied buffer.", "ECONNRESET": "Broken connection.", "ECANCELED": "Current coroutine was canceled.", "ENOTSUP": "The handle does not support this operation.", } errs = {} for e in fx["errors"]: errs[e] = standard_errors[e] errs.update(fx["custom_errors"]) errors = t/"None." if len(errs) > 0: errors = (t/'').vjoin([t/'* **@{e}**: @{desc}' for e, desc in sorted(errs.items())]) if fx["add_to_errors"]: errors |= t%'' | t/'@{fx["add_to_errors"]}' # Generate the manpage. page = t/""" # NAME @{fx["name"]} - @{fx["info"]} # SYNOPSIS ```c @{synopsis} ``` # DESCRIPTION @{description} # RETURN VALUE @{retval} # ERRORS @{errors} """ # Add EXAMPLE section, if available. example = "" if fx["protocol"] and fx["protocol"]["example"]: example = t/'@{fx["protocol"]["example"]}' if fx["example"]: example = t/'@{fx["example"]}' if example: page |= t%'' | t/""" # EXAMPLE ```c @{example} ``` """ # SEE ALSO section. # It'll contain all funrction from the same topic plus the functions # added manually. sa = [f["name"] for f in topics[fx["topic"]] if f["name"] != fx["name"]] if fx["has_deadline"]: sa.append("now") if fx["allocates_handle"]: sa.append("hclose") if fx["protocol"] and fx["protocol"]["type"] == "bytestream": sa.append("brecv") sa.append("brecvl") sa.append("bsend") sa.append("bsendl") if fx["protocol"] and fx["protocol"]["type"] == "message": sa.append("mrecv") sa.append("mrecvl") sa.append("msend") sa.append("msendl") # Remove duplicates and list them in alphabetical order. sa = list(set(sa)) sa.sort() #seealso = t/"" #for f in sa: # seealso += t/"**@{f}**(3) " seealso = (t%' ').join([t/'**@{f}**(3)' for f in sa]) page |= t%'' | t/""" # SEE ALSO @{seealso} """ with open(fx["name"] + ".md", 'w') as f: f.write(str(page))

Generally speaking, I am happy with the experiment.

I would like to highlight the following facts:

1. There's no DSL in the code whatsoever. Not even a simple one.
2. The source code is nicely indented.
3. The generated code is nicely indented.
4. There isn't a single escape sequence in the code.

On the other hand, there are some minor warts.

For example, when you want to join multiple strings in Python, you can do it this way:

result = ",".join("circle", "square", "triangle")

When you want to join multiple tiles though you need an extra pair of parentheses:

result = (t/",").join(t/"circle", t/"square", t/"triangle")

When you want to put one tile below another and separate them by a blank like you do it as follows:

result = a | t%"" | b

The empty line literal (t%"") looks too much like a vim command.

However, both of these problems are artifacts of hacking the tile support on top of existing Python language rather than having the tile primitive type supported by the language itself. If there was native support for tiles, the code would look like this:

result = t",".join(t"circle", t"square", t"triangle") result = a | w"" | b

Feb 21st, 2019

by martin_sustrik

WARNING: The claims about composability of the graceful shudown construct have been defeated in the discussion here. Or maybe not. Judge for yourself. However, most of the analysis in this article is still relevant.

Introduction

Within a structured concurrency world, graceful shutdown is a little understood, little addressed and confusing topic.

I've been fighting with it myself for almost a year (longer than with any other problem I can recall) and it haven't been a pleasant experience. If felt like running in circles. Too many contradictory requirements, to much complexity and too often a solution I've came up with turned out to be one of the solution I've already rejected in disguise.

Nathaniel Smith on structured concurrency forum helpfully pointed me to the discussion about graceful shutdowns in Trio and oh boy, does that discussion sound familiar.

Anyway, finally I've came up with a solution that's simple and works for all use cases. But it turns out it's one of those ideas that seem not to make sense until something clicks in one's mind. From there on though, they seem perfectly obvious, up to the point where one wonders how the others can possibly not get it.

In this post I'll try to explain.

The use case

The classic use case for graceful shutdown is a web server. It's handling many connections from the clients. When it's being shut down, you want it to happen in a controlled manner so that clients don't end up with half-loaded pages and such. Specifically, you want to stop the accept loop to make sure that no new connections are being made. Then you want to wait for say 10 seconds to give existing connections time to finish their work. Finally, when the time is up, you want to cancel any remaining connections and exit the process.

Of course, the nature of the workload may vary. Web server needs to shut down connections. CRUD application needs to shut down its connection to database. Yet different application may want to save data to disk before exiting.

What's graceful shutdown, anyway?

Graceful shutdown intuitively seems to be some kind of cancellation. Here, I would like to compare it to the classic hard cancellation and argue that graceful shutdown isn't anything like it.

Let's have a look at hard coroutine cancellation first. The code of the coroutine looks like this:

rc = send(s, "Hello!"); if(rc == ECANCELED) return; rc = recv(s, &msg); if(rc == ECANCELED) return; rc = sleep(60); if(rc == ECANCELED) return;

When parent coroutine hard-cancels a child coroutine, a blocking call within the coroutine will immediately return an error (or, in high-level languages, it will throw an exception) and the coroutine will exit.

Think of hard cancellation as "asking the coroutine to return immediately". In reality, it may do some bookkeeping tasks before exiting (freeing the allocated memory etc.) but form the perspective of this article, thinking of hard cancellation as eliciting immediate "return" in the coroutine is a pretty good approximation.

Now have a look at graceful shutdown.

It's very different. By saying "shut down gracefully" we are asking the coroutine to finish whatever it was doing and start doing different kind of work, namely it's shutdown business logic. So, for example, while during its normal lifetime it may be receiving data from the network and writing them to the database, after graceful shutdown request it may want to exchange terminal handshakes with the network peers and start flushing any in-memory data to the database.

Not only is graceful shutdown not immediate. It can be arbitrarily long or even infinite. Consider the case when you are trying to exchange a terminal handshake with a network peer. If the peer is not responding, the graceful shutdown will never finish.

More on the nature of graceful shutdown: In-band vs. out of band

Consider a case where the parent coroutine sends the child coroutine a stream of tasks to process. It does so via a channel. It may send DoA, DoB and DoC requests. Then it hard-cancels the child. Hard cancellation is supposed to be immediate and so the child coroutine exits even though DoA, DoB and DoC haven't been processed yet.

Does the graceful shutdown work is a similar way? No. Graceful shutdown means "finish whatever you are doing but take your time and be careful; don't break the business logic". During graceful shutdown you definitely don't want to drop the requests on the floor. So, instead of sending an out-of-band signal, like hard cancel did, you just write an application-defined STOP message to the channel and leave the child coroutine to do it's work. After it processes DoA, DoB and DoC, it will eventually receive the STOP message. At that point it will do the terminal handshakes, it will flush data to disk and so on. Finally, it will exit.

The point I am trying to make here is that while hard-cancel signal travels necessarily out-of-band, the graceful shutdown signal must be, equally necessarily, passed in-band.

And when you think about it a little bit more, "out-of-band" means that the communication channel is created and managed, invisibly to the user, by the language runtime (or a structured concurrency library, such as Trio or libdill) while "in-band" means that the communication channel is business-logic-specific and that it is defined and managed by the application itself.

In consequence, language runtime (or the library) has no way of knowing how to send a graceful shutdown to the child coroutine, even if it wanted to. The coroutine is listening on an application-specific channel. It may be channel A or channel B. It may be listening on both. Or maybe it's listening on channel A up to some point in its lifecycle and then starts listening on channel B. Or maybe it even gets the graceful shutdown request over the network (the latter is common during terminal handshake in network protocols). The language runtime just doesn't know.

Ergo, sending a graceful shutdown request cannot possibly be a feature of the language. It must be done manually by the application.

Even more on the nature of graceful shutdown: Cancellation points

The rules of hard cancellation are strict: Once a coroutine has been hard-canceled the very next blocking call will immediately return ECANCELED. In other words, every single blocking call is a cancellation point.

Graceful shutdown is different. It allows the coroutine to exit only at special points in the workflow where the business logic is in consistent state.

Consider the case of an RPC server. It reads a message from the peer, does some computation, then sends a reply to the peer.

Graceful shutdown can terminate the coroutine only when it is idling and waiting for a new request.

It we allowed graceful shutdown to terminate the coroutine while it is trying to send the reply, it would mean that a request could go unanswered. And that doesn't deserve to be called "graceful shutdown".

Let's summarize

What we learned so far is that hard cancellation:

1. Is triggered via an invisible communication channel created by the language runtime.
2. It manifests itself inside the target coroutine as an error code (ECANCELED in libdill) or an exception (Cancelled in Trio).
3. The error (or the exception) can be returned from any blocking call.
4. In response to it, the coroutine is not expected to do any application-specific work. It should just exit.

Graceful shutdown, on the other hand:

1. Is triggered via an application-specific channel.
2. Manifests itself inside the target coroutine as a plain old message.
3. The message may only be received at specific, application-defined points in the coroutine.
4. In response to it, the coroutine can do arbitrary amount of application-specific work.

Well, that seems to be it.

Hard cancellation is fully managed by the language. Graceful shutdown is fully managed by the application. From the language designer's perspective there's nothing to see here. We should just move on and forget about the entire business.

Or should we?

Keeping graceful shutdown in check

Well, it turns out that the parent coroutine often doesn't want graceful shutdown to take forever. It's willing to give it ten seconds to finish, but if that doesn't happen, so be it, c'est la vie, it will just hard-cancel it.

void main() { ch = channel(); b = bundle(); // a.k.a. scope or nursery bundle_go(b, worker(ch)); ... send(ch, STOP); // ask the child to shut down gracefully sleep(10); bundle_cancel(b); // hard-cancel the child if it haven't finished yet }

You may see the problem with this code. It works OK, but it always waits for 10 seconds, even when the child exits immediately. And that sounds like a waste.

So, after all, the language can do something to help with the graceful termination. It can provide a primitive to avoid the ugly non-negotiable sleep() step.

The proposal

First, let me be clear. I don't propose to add any new concept to the existing structured concurrency model. Not even the concept of "graceful shutdown". We'll do with the existing concepts of coroutine, bundle (scope, nursery) and plain old hard cancellation.

Second, I propose to add a timeout parameter to the traditional bundle_cancel() method:

bundle_cancel(b, 10);

The semantics are as follows: The function will wait for 10 seconds for all coroutines in the bundle to finish. If they all do before the timeout expires, the function will exit immediately as the last coroutine finishes. If they don't, it will hard-cancel all the remaining coroutines when the time is up.

In either case, when the function exits there are no more coroutines running in the bundle. If fact, the bundle itself doesn't exist any more.

The function also happens to be a blocking function and thus it behaves as any other blocking function. When the parent coroutine itself is hard-canceled bundle_cancel() hard-cancels all the coroutines in the bundle and returns ECANCELED error to the caller.

NOTE: With timeout set to zero, this is equivalent to the plain old hard cancellation. This has some pretty obvious implications for backward compatibility.

The examples

I am aware that such a trivial construct doesn't look like it's up to the task of dealing with the hydra of graceful shutdown.

So, to convince you, let us look how it fares in the wild.

The trivial case

In this scenario the child coroutine does some work and exits.

coroutine void worker(socket_t s) { int rc = send(s, "Hello, world!"); if(rc == ECANCELED) return; message_t msg; rc = recv(s, &msg); if(rc == ECANCELED) return; return; } int main(void) { socket_t s = create_connected_socket(); bundle_t b = bundle(); bundle_go(b, worker(s)); sleep(60); bundle_cancel(b, 10); return 0; }

Note how the worker coroutine knows nothing about graceful shutdown. It looks exactly the same way as it would if there was only hard cancellation. Yet the parent coroutine still manages to shut it down gracefully!

If the worker manages to finish its work in 10 seconds, the program will exit immediately. If it doesn't manage to finish in time, say if it is stuck in the recv() call, it will get hard-canceled after 10 seconds and the program will exit.

Infinite loop

This example deals with the question of what happens if the child coroutine never finishes.

coroutine void worker(socket_t s) { while(1) { int rc = send(s, "Hello, world!"); if(rc == ECANCELED) return; message_t msg; rc = recv(s, &msg); if(rc == ECANCELED) return; } } int main(void) { socket_t s = create_connected_socket(); bundle_t b = bundle(); bundle_go(b, worker(s)); sleep(60); bundle_cancel(b, 10); return 0; }

The code is identical to the previous example, except for the forever loop that was added to the worker. And once again, the child doesn't do anything specific to enable graceful shutdown.

You may feel like this example is a bit pointless. The worker coroutine runs at full speed throughout the entire grace period and then it gets hard-cancelled. It's hardly the graceful shutdown of your dreams.

But while that may be true, I am trying to make a point here. And the point is: The parent doesn't have to care.

The parent doesn't have to care whether the coroutine supports a graceful shutdown or not. It doesn't have to worry about how long the shutdown would take. It doesn't have to care whether there are infinite loops or not. It doesn't care if the worker behaves in a pointless way as shown in the example above.

The programmer doesn't have to go and inspect all the coroutines he is using to find out how the graceful shutdown would behave.

He just calls bundle_close(b, 10) and he is guaranteed that the coroutine will be dead in at most 10 seconds.

Graceful shutdown-aware coroutine

As already mentioned, the language has no concept of graceful shutdown.

Therefore, the parent coroutine has to send the graceful shutdown signal to the worker coroutine manually. It does so via a channel.

coroutine void worker(socket_t s, channel_t ch) { while(1) { message_t msg; int rc = recv_from_socket_or_channel(s, ch, &msg); if(rc == ECANCELED) return; if(rc == FROM_CHANNEL) goto graceful_shutdown; if(rc == FROM_SOCKET) { // process msg here } rc = send(s, "Hello, world!"); if(rc == ECANCELED) return; } graceful_shutdown: ... // do the entire graceful shutdown thing here } int main(void) { socket_t s = create_connected_socket(); channel_t ch = channel(); bundle_t b = bundle(); bundle_go(b, worker(s, ch)); sleep(60); send(ch, "STOP"); // ask for graceful shutdown bundle_cancel(b, 10); // give it at most 10 seconds to finish return 0; }

You may have noticed that I've been cheating here. recv_from_socket_or_channel() function hides a lot of complexity and to be fair, I am not aware of a language that has good API for this kind of thing. However, it is unrelated to the topic of graceful shutdown, so I'll let it be for now.

That being said, what I wanted to show here is how the cancellation points for graceful shutdown checkpoints are application-defined. In particular there's only one place in the workflow where STOP message can be received.

The application is basically an RPC server, waiting for a message from a client and then sending a "Hello, world!" reply. And we certainly don't want the client to miss their hello world message! Therefore, the only valid point to deal with the graceful shutdown is when the coroutine is idling, waiting for a new request.

If we checked for STOP before the send call, the coroutine would exit and RPC would end up unanswered. It would a large stretch to call that "graceful shutdown".

Graceful shutdown triggered by a network peer

I've already mentioned that it's entirely up to the application to decide how the graceful shutdown signal is delivered to the coroutine. One specific case of this is when the signal comes from the outside.

It's a common scenario with network protocols. Each peer can ask for the connection to be shut down. For example, with WebSockets, each party can send a special CLOSE message to inform the peer they want to shut down. The peer should respond by sending CLOSE message of its own.

The following code is a server that responds to CLOSE message from the client.

coroutine void worker(socket_t s) { while(1) { message_t msg; int rc = recv(s, &msg); if(rc == ECANCELED) return; if(msg == "CLOSE") break; rc = send(s, "Hello, world!"); if(rc == ECANCELED) return; } int rc = send(s, "CLOSE"); if(rc == ECANCELED) return; return; } int main(void) { socket_t s = create_connected_socket(); bundle_t b = bundle(); bundle_go(b, worker(s)); sleep(60); bundle_cancel(b, 10); return 0; }

Note how this interacts with the server-initiated graceful shutdown. The worker coroutine replies to CLOSE messages in a straightforward way. On the other hand, when the server is being shut down it stops accepting new connection (left as an exercise for the reader) and gives existing connections 10 seconds to exit. If they don't they'll get hard-canceled.

There's a missing piece of course. When server initiates the graceful shutdown, the worker should send unsolicited CLOSE message to the client to inform it that it wants to shut down the connection and then it should wait for CLOSE in reply.

So here's the full implementation:

coroutine void worker(socket_t s, channel_t ch) { while(1) { message_t msg; int rc = recv_from_socket_or_channel(s, ch, &msg); if(rc == ECANCELED) return; if(rc == FROM_CHANNEL) goto server_initiated_shutdown; if(rc == FROM_SOCKET) { if(msg == "CLOSE") goto peer_initiated_shutdown; } rc = send(s, "Hello, world!"); if(rc == ECANCELED) return; } server_initiated_shutdown: int rc = send(s, "CLOSE"); if(rc == ECANCELED) return; // drop all incoming messages until CLOSE is encountered while(1) { message_t msg; rc = recv(s, &msg); if(rc == ECANCELED) return; if(msg == "CLOSE") break; } return; peer_initiated_shutdown: int rc = send(s, "CLOSE"); if(rc == ECANCELED) return; return; } int main(void) { socket_t s = create_connected_socket(); channel_t ch = channel(); bundle_t b = bundle(); bundle_go(b, worker(s, ch)); sleep(60); send(ch, "STOP"); // ask for graceful shutdown bundle_cancel(b, 10); // give it at most 10 seconds to finish return 0; }

Graceful shutdown is composable

Now that we are done with examples, let's look at the thing from a conceptual point of view.

I've already mentioned that parent coroutine doesn't care about specifics of the shutdown mechanism of the child coroutine. It doesn't care if the child has infinite loops, it doesn't care whether it has application-level support for graceful shutdown and so on.

But there's one more thing it doesn't care about. Guess what. It doesn't care if the child has children of its own.

It also doesn't care whether it is already in the process of graceful shutdown, initiated, for example, by a remote peer, by an internal failure or maybe by some kind of coroutine-wide timeout.

This enforces strict isolation between coroutines and makes the graceful shutdown composable.

Let's illustrate the point on a concrete example:

coroutine void nested_worker(message_t msg) { // process the message here } coroutine void worker(socket_t s, channel_t ch) { bundle_t b = bundle(); while(1) { message_t msg; int rc = recv_from_socket_or_channel(s, ch, &msg); if(rc == ECANCELED) goto hard_cancellation; if(rc == FROM_CHANNEL) goto graceful_shutdown; if(rc == FROM_SOCKET) { bundle_go(b, nested_worker(msg)); } rc = send(s, "Hello, world!"); if(rc == ECANCELED) goto hard_cancellation; } graceful_shutdown: rc = bundle_cancel(b, 20); // cancel the nested workers with 20 second grace period if(rc == ECANCELED) return; return; hard_cancellation: rc = bundle_cancel(b, 0); // cancel the nested worker immediately if(rc == ECANCELED) return; return; } int main(void) { socket_t s = create_connected_socket(); channel_t ch = channel(); bundle_t b = bundle(); bundle_go(b, worker(s, ch)); sleep(60); send(ch, "STOP"); // ask for graceful shutdown bundle_cancel(b, 10); // give it at most 10 seconds to finish return 0; }

Note that when the server is being shut down it allows only for 10 second grace period. However, worker(), when being gracefully shut down gives its children 20 second grace period. That sounds like it's going to be a headache, but it turns out it is not.

When the 10 second grace period granted by the main() function expires it hard-cancels the worker(). The worker may be still waiting for its children to finish. But we don't care. Hard-cancelling the worker() means that "bundle_cancel(b, 20)" will immediately hard-cancel all the nested workers and exit with ECANCELED error (double-check the semantics of bundle_cancel() as proposed above).

All in all, we get the intuitive behaviour: The server shuts down in 10 seconds and hard-cancels all the children and grand-children that were still running at the time.

Implications for higher level languages

It's obvious that in higher level languages a lot of the boilerplate code can be avoided. For example, with "Canceled" exception all the "if(rc == ECANCELED) return;" lines just go away.

However, how to wrap the entire thing into a higher-level level constructs, such as automatic scopes, is a probem yet to be solved.

Feb 16th, 2019

by martin_sustrik

There are structured-concurrency-related efforts going on for different programming languages but the entire effort is kind of scattered, without people being aware of each other and speaking to each other.

If we had a common forum, we could share the use cases, the problems, the ideas and the solutions. Each of us, irrespective of which language they are working with, could benefit from this common pool of knowledge…

And here we go!

The forum now exists. Thanks to Nathaniel Smith for setting it up!

And here's my kick-off post on the forum.

Enjoy!

Feb 10th, 2019

by martin_sustrik

I've written about the cost of abstraction before. Once you are in the IT industry for couple of decades and once you've read couple of millions lines on legacy code you become healthily suspicious of any kind of abstraction. Not that we can do without abstraction. We need it to be able to write code at all. However, each time you encounter an abstraction in the code that could have been avoided you get a little bit sadder. And some codebases are sadder than Romeo and Juliet and King Lear combined.

Remember reading an unfamiliar codebase the last time? Remember how you've thought that the authors were a bunch of incompetent idiots?

People may argue that this is because legacy stuff is necessarily convoluted, but hey, at that point you were just skimming through the codebase and you weren't understanding it deep enough to tell your typical enterprise legacy monstrosity from a work of an architectural genius. The reason you were annoyed was because you were overwhelmed by the sheer amount of unfamiliar abstraction. (To prove that, consider what was your opinion of the codebase was few months later, after getting familiar with it. It looked much better, no?)

Keep that feeling in mind. Think of it when writing new code. How will a person who doesn't know first thing about this codebase feel when reading it?

The options are not palatable. Either you try to be clever, use abstraction a lot and they'll think you are a moron. Or you get rid of all unnecessary abstraction. You'll make their life much less frustrating but they'll think you are some kind of simpleton. (And they'll probably refactor the code to make it look more clever.)

I want to give a very basic example of the phenomenon.

Imagine that the requirements are that your program does A, B, C, D and E, in that order.

You can do it in the dumbest possible way:

void main() { // Do A. ... // Do B. ... // Do C. ... // Do D. ... // Do E. ... }

Or maybe you notice that B, C and D are kind of related and comprise a logical unit of work:

void foo() { // Do B. ... // Do C. ... // Do D. ... } void main() { // Do A. ... foo(); // Do E. ... }

But C would probably be better off as a stand-alone function. You can imagine a case where somewhene would like to call it from elsewhere:

void bar() { // Do C. ... } void foo() { // Do B. ... bar(); // Do D. ... } void main() { // Do A. ... foo(); // Do E. ... }

Now think of it from the point of view of casual reader, someone who's just skimming through the code.

When they look at the first version of the code they may thing the author was a simpleton, but they can read it with ease. It looks like a story. You can read it as if it were a novel. There's nothing confusing there. The parts come in the correct order:

A B C D E

But when skimming through the refactored code that's no longer the case. What you see is:

C B D A E

It's much harder to get the grip of what's going on there but at least they'll appreciate author's cleverness.

Or maybe they won't.

January 27th, 2019

by martin_sustrik

Generally, I don't tend to give talks (the last one I gave was six or seven years ago) but this time I am going to make an exception. I am going to give a short talk about structured concurrency as FOSDEM 2019.

If you are interested in the topic, meet me in Brussels on February 2nd!

Details about the event can be found here.

If you have no idea what structured concurrency is about, good intro article can be found here.

EDIT: The video of the talk can be found here.

by martin_sustrik