1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
|
# Copyright 2017 Yahoo Holdings. Licensed under the terms of the Apache 2.0 license. See LICENSE in the project root.
namespace=vespa.config.storage
## FILE LAYOUT PARAMETERS
## Number of directories per level to spread files across
dir_spread int default=256 restart
## Number of directory levels
dir_levels int default=1 restart
## FILE SIZE PARAMETERS
## Minimum number of meta data entries in one slotfile. When creating new
## files or resizing files, enforce it to contain at least this many meta
## entries. Set to 512 by default, using 20544 bytes total for metadata in
## new files.
minimum_file_meta_slots int default=512
## Maximum number of entries in one slotfile. File must be split before
## accepting more data
##
## Default ensure meta data is less than 512 kB.
maximum_file_meta_slots int default=13106
## Minimum size of header block. At least this amount of header space will
## be available in new or resized files. For 512 documents, this will be
## 200 bytes per document.
minimum_file_header_block_size int default=102848
## Maximum header block size (in bytes). Since the whole meta data list and
## header block needs to be read into memory for some operations, a limit
## can be set for the header block, to avoid consuming too much memory.
##
## Default is set high, as we dont configure it automatically right now, so we
## would rather people configure it down than up.
maximum_file_header_block_size int default=33554432
## Minimum size of a single slotfile. When creating or resizing files, they
## will never become smaller than this. Default of 1 MB, will be 1807 byte
## per doc if we use all 512 meta data entries set as default min meta
## entries.
minimum_file_size int default=1048576
## Maximum size of a single slotfile. File must be split before accepting
## more data. Will return file full errors.
##
## Default is set high, as we dont configure it automatically right now, so we
## would rather people configure it down than up.
maximum_file_size int default=268431360
## When creating new files, always create files as a multiplum of this size.
## Should be set to the block size on the disk, or possibly a multiplum of
## that, though we know no advantage of setting it to more than the block
## size. If you want more free space in files, you should rather adjust that
## with growfactor or min file sizes.
file_block_size int default=4096 restart
## CONSISTENCY PARAMETERS
## If true, fsync after all disk operations, to ensure no dirty OS file
## cache afterwards. This is only useful if using cached IO, which is not
## recommended to start with.
fsync_after_each_operation bool default=false restart
## Time period to keep all updates (given in seconds). One can revert any
## operation done within this time.
revert_time_period int default=300
## If a remove is older than the reverttimeperiod, the document it is
## removing may be removed from the file. There are a few advantages to
## keeping removes or a bit longer though. If you use this copy to
## synchronize another copy of data, having the remove entry makes it easy
## to detect that you should delete this entry from the other data copy.
## This is useful for internal synchronization of files within VDS if you
## use multiple copies, or for partial recovery or BCP situations. To
## guarantuee consistency in such situations, a data destination that have
## been unavailable for longer than this amount of time, should have its
## data cleared before being set into the system again. This value is given
## in seconds, with the default being one week
keep_remove_time_period int default=604800
## PERFORMANCE PARAMETERS
## Number of threads to use for each mountpoint. VDS needs memory per thread
## to perform disk operations, so increasing this number will increase
## memory usage, but it will also make the disk queue on a given disk be
## able to be larger, such that the disk can choose operations to optimize
## seek time.
## See benchmarks for performance/memory tradeoff.
threads[].lowestpri int default=255 restart
## Pause operations (and block new ones from starting) with priority
## lower than this value when executing operations with higher pri than
## min_priority_to_be_blocking
max_priority_to_block int default=255 restart
min_priority_to_be_blocking int default=0 restart
## When reading a slotfile, one does not know the size of the meta data
## list, so one have to read a static amount of data, and possibly read more
## if one didnt read enough. This value needs to be at least 64 byte to read
## the initial header stating the true size of the meta data list.
initial_index_read int default=61440
## When we need to read (or write) multiple entries in a file where we can
## either read a big enough section to cover all of them. But at some
## distance between the areas we need, it becomes beneficial to do multiple
## reads rather than to read over them. This setting set how many sequential
## bytes we dont need that we allow to be read/written in order to join two
## logical IO operations together in the application. Setting this low might
## be ok if system calls are cheap and we manage to queue up next IO
## operation in time such that the disk dont need to spin an extra round.
## Setting it high will make the disk more likely to process them together,
## but the time used to read/write the gap might have been used to do
## something useful instead.
maximum_gap_to_read_through int default=65536
## Size of the cache.
cache_size long default=1073741824 restart
## Soft limit on bytes used in cache for meta data, written as a percentage
## of the total
cache_size_meta_percentage long default=33 restart
## Soft limit on bytes used in cache for meta data, written as a percentage
## of the total
cache_size_header_percentage long default=33 restart
## Soft limit on bytes used in cache for meta data, written as a percentage
## of the total
cache_size_body_percentage long default=34 restart
## When a disk is fuller than this factor, we will not allow more data to be
## written to the system, unless this data is written in order to reduce
## storage consumption, such as resizing files to become smaller or add
## meta entries to write remove entries into. This value is set high as
## default as we expect a lot of users to have formatted their disks to
## already reserve 8% of the data to root user which is often default. We
## suggest using 0% reserved for root, and rather set this parameter lower
## to reserve space. That way, VDS have more space available in worst case
## in order to resize files to become smaller.
disk_full_factor double default=0.98 restart
## We don't allow disk moves to any disk that is fuller than this. This is a
## safeguard to avoid disk movement to fill up a disk.
disk_full_factor_move double default=0.80 restart
## The grow factor sets how much free space to add to a file when we resize
## it, whether we are making it smaller or larger. If the space we need
## after the operation triggering the resize is X, then the file will be
## resized to be of size X * growfactor. In a growing system with no
## deletes, a growfactor of 2 will make the files have 25% free space on
## average. Reducing it to 1.5 will reduce the average free space to 16.7%.
grow_factor double default=2.0
## If files fall below this fill rate, resize file to become smaller.
## Note that this parameter is tightly coupled with the growfactor. For
## instance, with a growfactor of 2.0, a file will contain 50 % free space
## after a resize. If the min fill rate then is 0.50, that means that if a
## single doc is deleted from this file, we need to resize it to
## become smaller straight away.
min_fill_rate double default=0.1
## When creating/resizing slotfiles, one uses average document sizes to
## decide how much free space to add to metadata, header and body portions.
## This option can be used to allocate extra free space to meta data in
## order to reduce the chance of the file needing resize due to lack of free
## meta data entries.
overrepresent_meta_data_factor double default=1.2
## When creating/resizing slotfiles, one uses average document sizes to
## decide how much free space to add to metadata, header and body portions.
## This option can be used to allocate extra free space to header data in
## order to reduce the chance of the file needing resize due to lack of
## header block space.
overrepresent_header_block_factor double default=1.1
## If non-empty, stores remove entries in such a way that the cluster can
## be downgraded to 5.0 without causing incompatibilities.
## Do not use this unless you know exactly why you are doing it!
store_50_backwards_compatible_remove_entries_with_doctype string default=""
|
