Was ist ccbot?
indexiert und analysiert Inhalte von Webseiten.
Der Webrobot ccbot indexiert und analysiert Inhalte von Webseiten. Er zeigt sich meistens mit der IP Adresse 100.26.46.245 und unter Verwendung des User Agent CCBot/2.0 (https://commoncrawl.org/faq/). Mit 0.0931% Marktanteil ist ccbot auf Platz 35 der aktivsten Webrobots im Internet.
„Common Crawl is a 501(c)(3) non-profit organization dedicated to providing a copy of the internet to internet researchers, companies and individuals at no cost for the purpose of research and analysis."
— Offizielle Beschreibung des Betreibers
robots.txt – ccbot blockieren
Füge diese Zeilen in deine robots.txt ein, um ccbot den Zugriff auf deine Website zu verwehren:
robots.txt
User-agent: ccbot
Disallow: /Du kannst den Zugriff auch gezielt einschränken, statt ihn komplett zu blockieren:
robots.txt (selektiv)
User-agent: ccbot
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow: /IP-Adressen 820 bekannte IPs
Diese IP-Adressen wurden bisher von ccbot verwendet:
100.26.46.245
107.20.20.39
107.20.34.144
107.22.139.217
107.23.176.162
174.129.103.100
174.129.109.148
174.129.127.214
174.129.135.89
174.129.151.95
174.129.167.207
174.129.64.33
174.129.72.165
174.129.74.186
174.129.92.127
174.129.93.155
174.129.96.175
18.204.48.241
18.205.192.78
18.208.211.150
18.212.222.217
18.212.250.179
18.212.83.37
18.212.85.149
18.215.161.19
18.232.140.235
18.234.236.14
184.73.104.82
184.73.110.237
184.73.110.238
184.73.112.180
184.73.119.213
184.73.126.70
184.73.13.66
184.73.18.109
184.73.3.107
184.73.40.21
184.73.72.163
184.73.8.127
184.73.90.167
23.20.240.193
23.20.245.192
23.20.91.134
23.21.38.201
23.22.136.56
23.22.17.192
23.22.173.58
23.22.19.253
23.22.194.120
23.22.2.150
23.22.20.49
23.22.204.36
23.22.44.0
23.22.45.86
23.22.46.195
23.22.53.212
23.22.59.252
23.22.79.235
23.23.12.202
23.23.31.73
23.23.36.47
23.23.46.20
23.23.47.38
23.23.53.177
23.23.57.144
23.23.57.182
23.23.61.241
23.23.62.93
23.23.8.131
23.23.9.5
3.208.22.127
3.227.10.125
3.80.128.196
3.80.223.123
3.80.55.37
3.80.85.76
3.81.2.11
3.81.28.94
3.81.73.233
3.82.44.44
3.83.229.176
3.84.186.122
3.85.143.239
3.86.63.79
3.88.220.93
3.89.255.232
3.90.108.129
3.90.205.8
3.90.56.208
3.91.157.213
3.94.100.103
3.94.196.192
3.95.23.35
34.203.213.116
34.203.225.78
34.203.28.212
34.226.208.152
34.228.41.66
34.228.43.110
34.228.55.57
34.229.175.129
34.229.194.198
34.229.97.16
34.236.187.117
34.238.194.166
34.239.164.150
35.172.201.102
35.175.200.4
50.16.100.60
50.16.112.199
50.16.118.223
50.16.165.62
50.16.173.192
50.16.24.12
50.16.31.61
50.16.40.195
50.16.41.165
50.16.47.14
50.16.49.108
50.16.68.229
50.16.75.46
50.16.78.128
50.16.78.31
50.16.84.67
50.16.98.1
50.17.167.146
50.17.86.12
50.19.169.37
50.19.171.113
50.19.172.0
50.19.206.49
52.23.215.188
52.90.59.110
52.91.185.49
52.91.245.237
54.144.206.73
54.144.210.245
54.144.231.243
54.144.237.48
54.144.240.110
54.144.246.252
54.144.251.157
54.144.251.8
54.144.254.112
54.144.254.174
54.144.255.152
54.145.132.82
54.145.141.127
54.145.144.101
54.145.145.2
54.145.158.70
54.145.161.33
54.145.162.230
54.145.164.64
54.145.165.174
54.145.166.247
54.145.167.92
54.145.168.90
54.145.172.149
54.145.173.147
54.145.173.57
54.145.173.91
54.145.176.120
54.145.179.206
54.145.180.142
54.145.180.58
54.145.181.230
54.145.182.50
54.145.183.126
54.145.183.128
54.145.183.169
54.145.183.190
54.145.183.207
54.145.186.248
54.145.188.41
54.145.191.14
54.145.191.30
54.145.194.205
54.145.202.174
54.145.207.36
54.145.209.130
54.145.209.42
54.145.209.77
54.145.209.80
54.145.231.20
54.145.235.72
54.145.238.55
54.145.243.51
54.145.246.183
54.145.252.85
54.145.254.249
54.146.134.152
54.146.143.102
54.146.169.117
54.146.171.44
54.146.174.220
54.146.178.218
54.146.180.94
54.146.190.193
54.146.50.80
54.146.98.143
54.147.142.16
54.147.183.203
54.147.193.224
54.147.204.207
54.147.212.98
54.147.213.234
54.147.214.74
54.147.237.138
54.147.243.244
54.147.248.118
54.147.250.33
54.147.255.107
54.147.29.160
54.152.31.167
54.156.39.245
54.158.132.141
54.158.143.40
54.158.152.212
54.158.163.106
54.158.165.135
54.158.166.6
54.158.175.78
54.158.188.170
54.158.240.135
54.158.52.166
54.158.65.139
54.158.85.158
54.159.134.239
54.159.136.47
54.159.142.177
54.159.161.138
54.159.165.175
54.159.172.209
54.159.175.61
54.159.184.55
54.159.188.179
54.159.190.106
54.159.190.90
54.159.205.37
54.159.214.27
54.159.215.49
54.159.218.151
54.159.219.180
54.159.223.242
54.159.239.248
54.159.246.164
54.159.250.110
54.159.250.233
54.159.30.26
54.159.93.83
54.159.94.253
54.161.113.170
54.161.139.162
54.161.145.251
54.161.146.154
54.161.147.106
54.161.147.26
54.161.149.107
54.161.154.115
54.161.155.142
54.161.161.255
54.161.176.90
54.161.179.218
54.161.181.49
54.161.185.209
54.161.185.244
54.161.191.254
54.161.192.135
54.161.192.43
54.161.192.61
54.161.195.4
54.161.198.195
54.161.202.106
54.161.202.234
54.161.203.23
54.161.214.221
54.161.219.112
54.161.221.10
54.161.225.51
54.161.228.29
54.161.228.30
54.161.234.150
54.161.235.83
54.161.236.195
54.161.247.20
54.161.247.22
54.161.255.236
54.161.255.61
54.161.3.96
54.161.45.156
54.161.71.87
54.161.77.30
54.162.121.80
54.162.122.174
54.162.128.159
54.162.139.105
54.162.15.31
54.162.159.33
54.162.165.158
54.162.166.76
54.162.184.214
54.162.239.144
54.162.9.28
54.163.100.58
54.163.106.31
54.163.112.75
54.163.115.193
54.163.135.88
54.163.139.36
54.163.141.69
54.163.158.99
54.163.179.108
54.163.181.14
54.163.191.142
54.163.20.123
54.163.213.149
54.163.42.154
54.163.61.66
54.163.67.67
54.163.68.15
54.163.70.249
54.163.72.86
54.163.74.238
54.163.78.47
54.163.84.123
54.163.84.187
54.163.84.199
54.163.89.150
54.163.89.8
54.163.90.41
54.163.91.250
54.164.198.240
54.166.101.170
54.166.102.61
54.166.104.226
54.166.105.24
54.166.108.167
54.166.110.247
54.166.111.111
54.166.113.249
54.166.114.14
54.166.114.43
54.166.114.53
54.166.116.36
54.166.117.130
54.166.117.85
54.166.12.41
54.166.120.175
54.166.122.65
54.166.122.69
54.166.122.86
54.166.122.88
54.166.122.92
54.166.123.2
54.166.128.254
54.166.130.157
54.166.14.218
54.166.15.152
54.166.152.187
54.166.159.110
54.166.24.112
54.166.255.168
54.166.32.4
54.166.33.25
54.166.39.179
54.166.45.31
54.166.50.213
54.166.54.215
54.166.62.223
54.166.62.226
54.166.66.204
54.166.84.82
54.166.85.209
54.166.96.101
54.167.10.244
54.167.126.106
54.167.130.74
54.167.132.102
54.167.138.172
54.167.138.177
54.167.138.53
54.167.143.63
54.167.144.170
54.167.144.202
54.167.144.4
54.167.153.63
54.167.155.147
54.167.157.247
54.167.159.151
54.167.163.70
54.167.171.186
54.167.171.79
54.167.173.217
54.167.173.250
54.167.174.11
54.167.174.90
54.167.175.107
54.167.175.157
54.167.175.216
54.167.175.218
54.167.175.240
54.167.175.241
54.167.176.58
54.167.176.81
54.167.177.111
54.167.177.180
54.167.177.207
54.167.178.129
54.167.179.48
54.167.182.201
54.167.182.81
54.167.184.188
54.167.185.110
54.167.185.18
54.167.186.83
54.167.190.76
54.167.230.68
54.167.66.102
54.167.75.155
54.167.83.233
54.172.133.111
54.196.103.26
54.196.108.105
54.196.119.93
54.196.127.107
54.196.13.210
54.196.145.24
54.196.159.11
54.196.168.236
54.196.190.32
54.196.225.45
54.196.26.1
54.196.46.127
54.196.62.23
54.196.63.93
54.196.73.22
54.197.110.151
54.197.111.87
54.197.116.116
54.197.147.98
54.197.15.196
54.197.159.193
54.197.168.70
54.197.171.28
54.197.179.69
54.197.183.230
54.197.199.228
54.197.211.197
54.197.45.100
54.197.65.82
54.197.65.97
54.197.66.57
54.197.69.219
54.197.94.241
54.197.94.30
54.198.102.156
54.198.104.202
54.198.11.16
54.198.11.210
54.198.120.209
54.198.122.70
54.198.126.110
54.198.158.24
54.198.169.113
54.198.221.13
54.198.25.229
54.198.3.15
54.198.42.105
54.198.96.198
54.198.98.25
54.204.100.232
54.204.107.48
54.204.146.61
54.204.154.108
54.204.155.155
54.204.155.41
54.204.162.36
54.204.165.156
54.204.165.89
54.204.168.212
54.204.178.31
54.204.179.32
54.204.182.118
54.204.185.245
54.204.218.95
54.204.243.45
54.204.244.121
54.204.249.184
54.204.68.123
54.204.92.234
54.204.94.228
54.204.97.152
54.205.101.118
54.205.105.23
54.205.115.177
54.205.119.93
54.205.122.189
54.205.122.62
54.205.127.52
54.205.130.92
54.205.144.54
54.205.157.94
54.205.167.187
54.205.170.21
54.205.189.1
54.205.20.160
54.205.207.201
54.205.211.87
54.205.230.3
54.205.241.107
54.205.34.249
54.205.60.49
54.205.68.5
54.205.73.176
54.205.74.11
54.205.75.60
54.205.88.118
54.205.96.97
54.205.98.35
54.210.61.41
54.211.100.183
54.211.101.8
54.211.113.223
54.211.137.77
54.211.160.88
54.211.17.91
54.211.180.175
54.211.181.45
54.211.182.241
54.211.192.223
54.211.210.27
54.211.212.174
54.211.217.197
54.211.219.20
54.211.219.68
54.211.23.190
54.211.230.186
54.211.235.255
54.211.236.52
54.211.34.105
54.211.68.132
54.211.73.232
54.211.79.99
54.211.80.150
54.211.96.99
54.211.97.242
54.221.147.93
54.221.75.115
54.221.9.6
54.224.117.28
54.224.148.23
54.224.166.141
54.224.17.157
54.224.175.2
54.224.187.45
54.224.201.110
54.224.53.192
54.224.56.126
54.224.75.0
54.224.85.115
54.224.91.246
54.224.94.8
54.225.17.239
54.225.32.164
54.225.57.156
54.226.0.225
54.226.10.234
54.226.136.179
54.226.147.84
54.226.191.80
54.226.192.202
54.226.21.57
54.226.213.228
54.226.227.175
54.226.239.54
54.226.252.142
54.226.4.91
54.226.7.15
54.226.99.16
54.227.104.53
54.227.126.69
54.227.127.189
54.227.159.165
54.227.160.102
54.227.171.163
54.227.21.95
54.227.214.52
54.227.215.139
54.227.215.140
54.227.215.141
54.227.215.248
54.227.31.145
54.227.41.242
54.227.6.156
54.227.76.35
54.227.86.164
54.227.9.98
54.234.0.156
54.234.0.85
54.234.13.175
54.234.141.47
54.234.190.237
54.234.225.23
54.234.228.64
54.234.247.118
54.234.56.173
54.234.7.161
54.234.74.85
54.234.90.191
54.235.20.221
54.235.29.110
54.235.4.196
54.237.151.188
54.237.184.242
54.237.197.160
54.237.213.31
54.237.235.12
54.237.31.203
54.237.57.53
54.237.78.165
54.237.98.229
54.237.99.131
54.242.105.196
54.242.126.126
54.242.126.9
54.242.164.211
54.242.190.171
54.242.229.174
54.242.229.204
54.242.231.109
54.242.236.164
54.242.241.20
54.242.250.208
54.242.253.192
54.80.113.118
54.80.127.111
54.80.140.26
54.80.189.255
54.80.198.134
54.80.23.241
54.80.43.90
54.80.60.91
54.80.63.78
54.80.68.137
54.80.69.152
54.80.70.68
54.80.80.77
54.80.87.166
54.81.109.37
54.81.112.187
54.81.117.119
54.81.120.138
54.81.123.216
54.81.123.248
54.81.155.64
54.81.158.247
54.81.163.254
54.81.166.196
54.81.166.199
54.81.183.183
54.81.187.157
54.81.195.140
54.81.2.75
54.81.216.254
54.81.232.54
54.81.52.121
54.81.53.224
54.81.68.185
54.81.69.220
54.81.71.68
54.81.76.147
54.81.76.247
54.81.77.114
54.81.77.93
54.81.80.46
54.81.92.174
54.81.94.71
54.82.0.87
54.82.103.134
54.82.116.210
54.82.117.18
54.82.121.218
54.82.123.216
54.82.135.191
54.82.144.55
54.82.146.25
54.82.152.93
54.82.166.45
54.82.189.248
54.82.190.130
54.82.209.152
54.82.216.134
54.82.219.70
54.82.230.178
54.82.245.224
54.82.29.141
54.82.31.133
54.82.41.6
54.82.46.38
54.82.50.27
54.82.77.50
54.82.80.15
54.82.82.201
54.82.89.114
54.82.93.116
54.83.103.17
54.83.118.45
54.83.120.25
54.83.158.198
54.83.167.135
54.83.167.152
54.83.189.89
54.83.224.15
54.83.224.32
54.83.78.132
54.83.82.191
54.87.107.232
54.87.109.201
54.87.112.36
54.87.117.4
54.87.119.22
54.87.123.99
54.87.128.228
54.87.129.16
54.87.134.115
54.87.134.127
54.87.136.120
54.87.136.181
54.87.138.122
54.87.15.219
54.87.155.193
54.87.160.113
54.87.179.107
54.87.180.131
54.87.180.135
54.87.180.138
54.87.180.149
54.87.180.181
54.87.181.206
54.87.198.240
54.87.216.170
54.87.221.168
54.87.30.2
54.87.47.251
54.87.60.7
54.87.70.14
54.87.72.176
54.87.72.90
54.87.73.191
54.87.75.51
54.87.76.100
54.87.77.15
54.87.83.162
54.87.88.36
54.87.88.52
54.87.89.176
54.87.89.244
54.88.92.203
54.89.0.97
54.89.1.216
54.89.12.223
54.89.187.28
54.89.36.189
54.89.47.28
54.89.53.55
54.89.60.200
54.89.68.105
54.89.79.183
54.89.79.33
54.89.91.165
54.90.101.112
54.90.119.59
54.90.128.222
54.90.147.77
54.90.153.141
54.90.156.100
54.90.158.245
54.90.18.61
54.90.184.38
54.90.201.145
54.90.227.221
54.90.227.32
54.90.233.153
54.90.239.205
54.90.243.204
54.90.253.173
54.90.46.227
54.90.66.237
54.90.66.70
54.90.92.204
54.91.120.117
54.91.125.72
54.91.159.232
54.91.16.49
54.91.16.95
54.91.163.224
54.91.192.195
54.91.227.159
54.91.25.210
54.91.35.130
54.91.42.7
54.91.45.143
54.91.47.154
54.91.49.40
54.91.84.149
54.92.163.105
54.92.194.75
54.92.197.82
54.92.211.239
54.92.232.155
54.92.232.249
54.92.233.125
54.92.236.88
54.92.236.96
54.92.239.120
54.92.239.26
54.92.242.86
54.92.255.172
54.92.255.64
User Agents
Mit diesen User-Agent-Strings identifiziert sich ccbot:
CCBot/2.0 (https://commoncrawl.org/faq/)CCBot/2.0 (http://commoncrawl.org/faq/)
Es ist augenscheinlich, wenn ein Roboter, der zur Suchmaschine im Web gehört, unzählig viele IP-Adressen hat, unerwünscht ist, oder?
Vielleicht klaut dieser und gibt unlogische Ergebnisse heraus?
Warum ist der Crowler unbeliebt?