为大型数据文件每行只能产生id

4个主要思路：

1 单线程处理

2 普通多线程

3 hive

4 Hadoop

 

搜到一些參考资料

《Hadoop实战》的笔记-2、Hadoop输入与输出

TextInputFormat：文件偏移量：整行数据

可是这个偏移量，貌似是在一个文件的偏移，而不是全局。

 

Generate Auto-increment Id in Map-reduceJob

 

Generate unique customer id / insert uniquerows in hive

 

Need to add auto increment column in atable using hive

 

 

Here make sure that addition of annotation@UDFType(stateful = true) is required otherwisecounter value will not get increment in the Hive column, it will just returnvalue 1 for all the rows but not the actual row number.

 

最后我採取了用hive写udf的方案。

package hive.udf;/** * Licensed to the Apache Software Foundation (ASF) under one * or more contributor license agreements.  See the NOTICE file * distributed with this work for additional information * regarding copyright ownership.  The ASF licenses this file * to you under the Apache License, Version 2.0 (the * "License"); you may not use this file except in compliance * with the License.  You may obtain a copy of the License at * *     http://www.apache.org/licenses/LICENSE-2.0 * * Unless required by applicable law or agreed to in writing, software * distributed under the License is distributed on an "AS IS" BASIS, * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. * See the License for the specific language governing permissions and * limitations under the License. */import org.apache.hadoop.hive.ql.exec.Description;import org.apache.hadoop.hive.ql.exec.UDF;import org.apache.hadoop.hive.ql.udf.UDFType;/** * UDFRowSequence. */@Description(name = "row_sequence",    value = "_FUNC_() - Returns a generated row sequence number starting from 1")@UDFType(deterministic = false, stateful = true)//stateful參数是必要的public class UDFRowSequence extends UDF{  private int result;  public UDFRowSequence() {    result=0;  }  public int evaluate() {	  result++;    return result;  }}// End UDFRowSequence.java

 

本文作者：linger

本文链接：

版权声明：本文博主原创文章，博客，未经同意不得转载。