Instructions to run the competition

GDMC Judgement Guide

Thank you for volunteering to help us evaluate the different submission to the second, AI Settlement Generation competition in Minecraft. We appreciate your help. If there are any questions, feel free to comment directly here in the guide. 

This guide is designed to help you through the judgement process, and contains the following items: (there are lots of pictures, so its not 13 pages of text)

Overview1

Installing the Maps2

1- Locate your minecraft installation folder and open the Saves folder2

MacOS:2

Windows:3

2- Copy the maps into the Minecraft Saves folder4

3- Open Minecraft using version 1.12.25

4- Open the competition maps8

Evaluating the Entries9

The score sheet9

Scoring Guide:10

Towards human-like generation10

Score range10

Categories11

Adaptability:11

Functionality:12

Evocative Narrative:12

Maintaining Aesthetics:13

Some final notes:13


Overview

This competition is about building an AI that can build settlements (as in villages or towns) for a given Minecraft map. You can find a lot more detail on our homepage:

http://gendesignmc.engineering.nyu.edu/

At this point, several teams have submitted their AIs (in the form of python code, or as JAVA based mod). We have created three competition maps, and have then applied each team'steams AI to each of the three maps. YourYou job is now to evaluate these maps and ideally give some feedback. 

You should be able to download all relevant material, i.e. the maps and the score sheet from here:

https://www.dropbox.com/sh/2kvzuefusvl72tn/AAC0WmRO5tbosMCtD48_L0f8a?dl=0

The guide should walk you through all relevant steps, but basically, you need to install the maps, evaluate them, and then return a filled in score sheet to us via email, to 

gendesignMC@gmail.com

Please return the score no later than the 14th of June, 2019. 

Thank you for your help, and enjoy. 

The GDMC organization team

Installing the Maps

1- Locate your minecraft installation folder and open the Saves folder

MacOS:

 

On MacOS, the default path is ~/Library/Application Support/minecraft . However, the ~/Library folder is not shown on Finder by default. You can locate it on finder by opening the Go menu or pressing the keys Shift + Command + G

Once inside the minecraft folder, open the Saves folder

For more information on the Minecraft installation folder, see https://minecraft.gamepedia.com/.minecraft

Windows:

 

In Windows the saves (folders containing each map) are usually located at C:\Users\your_name_here\AppData\Roaming\.minecraft\saves

The quickest way to get there is to Press Win+R, then type %appdata%\.minecraftminecraftr, then press Ok.

Once inside the Minecraft folder, you can just click on the saves folder.

 

From this point on, the instructions for MacOS and Windows are similar. 

2- Copy the maps into the Minecraft Saves folder

The folder containing each map should be pasted directly to the saves folder, as shown in the picture:

3- Open Minecraft using version 1.12.2

It is important that you run the maps in Minecraft 1.12.2. Minecraft has unfortunately just updated to 1.13 – which contains a new terrain generator, which messes up the maps a bit if you load it in that version. So, to view the maps as intended, you have to take this extra step:

First, open Minecraft and go to Launch Options:

Click “Add new”, enter a name for the launch configuration, such as “GDMC (v 1.12.2)” and Save. The new configuration should appear among the launch options

Finally, click the Minecraft logo to go back to the starting screen and check if the recently created launch option is selected on Play. If it is not, click the little arrow icon on Play and select the desired configuration, then click Play to launch the game

For more information on changing the game version, see https://help.mojang.com/customer/portal/articles/1475923-changing-game-versions

4- Open the competition maps 

Make sure that the version on the bottom left corner of the screen is 1.12.2, then click Singleplayer

The maps should appear on the Select World menu. There are 18 files (e.g. Adrian_R2-1) containing the settlements generated by applying one of the participant’s filter to the basic maps. The R2 stands for round 2, and the numbers 1 to 3 indicate which of the basic maps the algorithm was applied to. There are 3 maps for each entry.

Note that the files are not necessarily in order, and there might maps unrelated to the competition on the menu depending on the contents of your saves folder

Evaluating the Entries

The score sheet

You will find the score sheet in the download folder, we included a .csv .xls and xlsx, so there should be a format that works for you. You only need to fill in one and send it back to us. 

After you look at all three maps, you should give each entry one score for each of the four scoring categories, adaptability, functionality, narrative and aesthetic. The final score is an average of the other scores, and should be computed automatically (unless you use the csv).

We also encourage you to leave comments to the competitors in the optional note field. We will compile those and forward them to the participants anonymously. Here you can tell them what you liked and disliked, and what you thought of their settlement generation algorithm. 

Scoring Guide:

In this section, we are explaining the ideas behind the different judging criteria. But keep in mind, we chose out judges based on the idea that you are experts in what you do, and we value your opinion, so in case of doubt, follow your intuition. 

Towards human-like generation

The main idea was to score the competition in a way that encourages participants to think about how to design AIs that can build Minecraft settlements similar to those built by humans. If you want to have an idea of what humans can do, have a look at our positive examples page:

http://gendesignmc.engineering.nyu.edu/rules

Or just have a look at this one gif here: 

https://thumbs.gfycat.com/ClumsyTanGibbon-mobile.mp4

To this end, we just thought about 4 different areas where we thought that there is still a difference between how well a computer can generate something like a Minecraft settlement, and how well  a human can do it. We discussed these in more detail in our paper:

https://arxiv.org/abs/1803.09853

But we will briefly reiterate the different categories, adaptability, functionality, narrative and aesthetic in this guide. 

Score range

The idea is to score the entries in each of those categories, to encourage participants to think on how to tackle each of those problems. Each category can score from 0 to 10. 

  • 0 points means that the resulting design shows no consideration of that particular criterion at all. 

  • 1 – 4 points means that there are some aspects in which the criterion is addressed

  • 5 points indicates a performance, in that area, that could be from a naive human. At this point, you would not be surprised if this was built by a human. 

  • 6 – 9 points indicates a expert level human performance, over a longer time, possibly a group effort. So, we are talking about a group of city planners and architects designing a Minecraft settlement over the course of a year. The higher end of the point scare here should mean a work that would possibly win a design price. 

  • 10 points – superhuman performance – this is so good, you are surprised if this could be generated by a dedicated group of expert humans. 

Categories

In our competition description we announced that we would supply the judges with questions to illustrate each category. This section will contain a short descriptions, and those guiding questions. Keep in mind that those questions are just illustrative, and not exhaustive. Each category is more complex, and there are ways to address these issues that go beyond those questions. Consequently, the questions are not a checkbox list – so if there are 5 questions, you should not just give 2 points for each. You should rather ask, how well did the AI perform here compared to a regular human. 

Adaptability:

Adaptability is about how well a given settlement fits into the map and the surrounding terrain. Is the design shaped by the given map, and is the map in turn shaped by the settlement. The challenge here is to not just generate something that can be put on any map, but to generate something that reflects the input (the map). 

– Do the structures in the settlement adapt to the terrain?

– Do the structure in the environment reflect the environment, i.e. usage of available material, adaptation to the biome?

– Does the settlement take advantage of terrain features or compensate for problems with the terrain?

– Are the settlements different in reaction to the different initial maps?

– Are there any other ways in how the settlement adapt to the given maps?

Functionality:

Settlements in general, and in Minecraft in particular, are not just aesthetic artefacts but also provide functionality in different forms. There a different kinds of functionality. First, there are issues that are not even dependent on Minecraft, like how easy is it to walk through the settlement, or how easy is it to navigate. There are also functions that are more closely tied to the game Minecraft, like keeping the monsters out, or providing enough light so monsters don’t spawn, or making food and crafting stations accessible to the player. This is definitely a category where the list of questions is not exhaustive – it is possible to provide extra affordances that we have not thought of. 

– Does the settlement provide protection from danger?

– Does it keep mobs from spawning?

– Does it keep mobs out?

– Protection from other environmental dangers?

– Is the settlement accessible to a player avatar in survival mode?

– Can you walk to everywhere?

– Does the settlement provide faster modes of transport?

– How easy is it to find your way around?

– Does the settlement provide the player with additional affordances?

– Does the settlement makes ressources easy to obtain?

– Is there an easy way to get food?

– Does the settlement provide functionality to the villagers?

– Does the settlement reflect the embodiment of the player avatar?

– Is it appropriately scaled?

Evocative Narrative:

In real life settlement tell stories about the people who build the, about their lifes, their culture, and their history. Settlements are also living testaments to the history that shaped them. When we look at settlements in reality, or even those design by humans for games, we get a sense of what this settlement is about, or how it was shaped over time. The challenge here is to automatically produce a settlement that evokes a distinct story – and ideally one that is adapted to the underlying input, the map. 

– Is the settlement evoking an interesting story?

– After looking at the settlement, could you give a short description of what this settlement is about that sets it apart from other settlements?

– Is it clear what the function of the settlement is?

– Does this function make sense in regards to the terrain and environment it is in? I.e. is the logging camp in a forest, the harbour town at the sea?

– Is the functionality of the settlement supporting this narrative function? I.e. does the fortified frontier settlement have functioning walls, is the farming village equipped with functioning fields?

– Does the final settlement give any indication of how the settlement developed?

– Is is possible to look at the settlement and imagine in what order things where build, or what stages the development of the settlement took?

– Is there an indication of the history of the settlement evident in the structure?

– Are the any convincing and consistent allusion to human cultures or specific points in history that the settlement is modelled after?

– Does the settlement have a culture – either fictional or historical, that is evident from the settlement?

– Do you know things about this culture just by looking at the settlement?

Maintaining Aesthetics:

The last criterion is not necessarily about beauty, but about a consistent look. The challenge here is not to only to produce something that looks like good design, but to do so, while also addressing the other challenges. It is somewhat simple to design a really great looking house, and just copy it down several times. But having an algorithm design houses and have them look well designed is a different question. This category contains a lot of elements that humans might get right without thinking about it, like building houses that are visually balanced, or well proportioned. In its corollary, this category is also about avoiding those jarring, strange artefacts that procedural design generates that would never be made by a human. 

– Does the settlement look good?

– Is there a consistent look to the settlement? Does it appear that all structures belong to the same settlement?

– Is there an appropriate level of variation in the existing structures?

– Are there any jarring features that make the settlement look unbelievable?

Some final notes:

Before you look at the settlement, you can also have a look at the maps given to the players. There are those without any entryX in the name. This might give you an idea of what was given to the algorithms. 

The game worlds are in creative mode, so you do not have to worry about dying or game play issues. While you can fly in the game, consider to also walk through the settlement, to experience it as a player would, or ask yourself, (if you have some Minecraft experience), how you would play in this settlement. 

Conclusion

Thank you for helping with the evaluation. If you have any further questions, feel free to email us, or leave a comment on this guide. 

Leave a Comment

Your email address will not be published. Required fields are marked *